Skip to content

F
faiss
Commit cf18101f authored by Lucas Hosseini's avatar Lucas Hosseini Committed by Matthijs Douze
Browse files
Options

Refactor makefiles and add configure script (#466) 

* Refactors Makefiles and add configure script.

* Give MKL higher priority in configure script.

* Clean up Linux example makefile.inc.

* Cleanup makefile.inc examples.

* Fix python clean Makefile target.

* Regen swig wrappers.

* Remove useless CUDAFLAGS variable.

* Fix python linking flags.

* Separate compile and link phase in python makefile.

* Add macro to look for swig.

* Add CUDA check in configure script.

* Cleanup make depend targets.

* Cleanup CUDA flags.

* Fix linking flags.

* Fix python GPU linking.

* Remove useless flags from python gpu module linking.

* Add check for cuda libs.

* Cleanup GPU targets.

* Clean up test target.

* Add cpu/gpu targets to python makefile.

* Clean up tutorial Makefile.

* Remove stale OS var from example makefiles.

* Clean up cuda example flags.
parent dcd83f95
Expand all Hide whitespace changes
Inline Side-by-side
Showing with 1051 additions and 359 deletions
.gitignore
View file @ cf18101f
/bin/
/c_api/bin/
/c_api/gpu/bin/
/makefile.inc
/tests/demo_ivfpq_indexing
*.swp
*.swo
*.o
*.a
*.dSYM
*.so
*.dylib
*.pyc
*~
/config.*
/aclocal.m4
/autom4te.cache/
/makefile.inc
/bin/
/c_api/bin/
/c_api/gpu/bin/
/tests/test
/tests/gtest/
FaissAssert.h
View file @ cf18101f
......@@ -56,7 +56,7 @@
#define FAISS_THROW_MSG(MSG) \
do { \
throw FaissException(MSG, __PRETTY_FUNCTION__, __FILE__, __LINE__); \
throw faiss::FaissException(MSG, __PRETTY_FUNCTION__, __FILE__, __LINE__); \
} while (false)
#define FAISS_THROW_FMT(FMT, ...) \
......@@ -65,7 +65,7 @@
int __size = snprintf(nullptr, 0, FMT, __VA_ARGS__); \
__s.resize(__size + 1); \
snprintf(&__s[0], __s.size(), FMT, __VA_ARGS__); \
throw FaissException(__s, __PRETTY_FUNCTION__, __FILE__, __LINE__); \
throw faiss::FaissException(__s, __PRETTY_FUNCTION__, __FILE__, __LINE__); \
} while (false)
///
......
INSTALL.md
View file @ cf18101f
[//]: # "**********************************************************"
[//]: # "** INSTALL file for Faiss (Fair AI Similarity Search **"
[//]: # "**********************************************************"
......@@ -76,6 +75,9 @@ dynamic libraries (useful for the Python wrapper).
Step 1: Compiling the C++ Faiss
===============================
TL;DR: `./configure && make && make install`
The CPU version of Faiss is written in C++11.
BLAS/Lapack
......@@ -90,18 +92,20 @@ thus does not need an include path.
There are several BLAS implementations, depending on the OS and
machine. To have reasonable performance, the BLAS library should be
multithreaded. See the example makefile.inc's for hints and examples
on how to set the flags.
on how to set the flags, or simply run the configure script:
`./configure`
To check that the link flags are correct, and verify whether the
implementation uses 32 or 64 bit integers, you can
`make tests/test_blas`
`make misc/test_blas`
and run
`./tests/test_blas`
`./misc/test_blas`
Testing Faiss
Building faiss
-------------
Once the proper BLAS flags are set, the library should compile
......@@ -109,6 +113,13 @@ smoothly by running
`make`
Then, in order to install the library and the headers, run
`make install`
Testing Faiss
-------------
A basic usage example is in
`demos/demo_ivfpq_indexing`
......@@ -117,6 +128,11 @@ it makes a small index, stores it and performs some searches. A normal
runtime is around 20s. With a fast machine and Intel MKL's BLAS it
runs in 2.5s.
To run the whole test suite:
`make test`
A real-life benchmark
---------------------
......@@ -128,11 +144,11 @@ http://corpus-texmex.irisa.fr/
and unzip it to the subdirectory `sift1M` at the root of the source
directory for this repository.
Then compile and run
Then compile and run the following (after ensuring you have installed faiss):
```
make demos/demo_sift1M
demos/demo_sift1M
make demos
./demos/demo_sift1M
```
This is a demonstration of the high-level auto-tuning API. You can try
......@@ -155,10 +171,10 @@ How it works
------------
The Python interface is provided via SWIG (Simple Wrapper and
Interface Generator) and an additional level of manual wrappers (in faiss.py).
Interface Generator) and an additional level of manual wrappers (in python/faiss.py).
SWIG generates two wrapper files: a Python file (`swigfaiss.py`) and a
C++ file that must be compiled to a dynamic library (`_swigfaiss.so`). These
SWIG generates two wrapper files: a Python file (`python/swigfaiss.py`) and a
C++ file that must be compiled to a dynamic library (`python/_swigfaiss.so`). These
files are included in the repository, so running swig is only required when
the C++ headers of Faiss are changed.
......@@ -253,7 +269,7 @@ Testing the GPU implementation
Compile the example with
`cd gpu; make test/demo_ivfpq_indexing_gpu`
`cd gpu; make tests/demo_ivfpq_indexing_gpu`
This produce the GPU code equivalent to the CPU
demo_ivfpq_indexing. It also shows how to translate indexed from/to
......@@ -265,7 +281,7 @@ Compiling the Python interface with GPU support
Given step 2, adding support of the GPU from Python is quite
straightforward. Run
`cd gpu; make py`
`cd python; make _swigfaiss_gpu.so`
The import is the same for the GPU version and the CPU-only
version.
......
Makefile
View file @ cf18101f
......@@ -4,184 +4,88 @@
# This source code is licensed under the BSD+Patents license found in the
# LICENSE file in the root directory of this source tree.
.SUFFIXES: .c .cpp .o
-include makefile.inc
SRC=$(wildcard *.cpp)
OBJ=$(SRC:.cpp=.o)
MAKEFILE_INC=makefile.inc
-include $(MAKEFILE_INC)
############################
# Building
PREFIX ?= /usr/local/
default: libfaiss.a
all: .env_ok libfaiss.a demos/demo_ivfpq_indexing
all: libfaiss.a libfaiss.$(SHAREDEXT)
py: _swigfaiss.so
libfaiss.a: $(OBJ)
ar r $@ $^
libfaiss.$(SHAREDEXT): $(OBJ)
$(CXX) $(SHAREDFLAGS) $(LDFLAGS) -o $@ $^ $(LIBS)
%.o: %.cpp
$(CXX) $(CPPFLAGS) $(CXXFLAGS) $(CPUFLAGS) -c $< -o $@
#############################
# Various
clean:
rm -f libfaiss.*
rm -f $(OBJ)
LIBOBJ=hamming.o utils.o \
IndexFlat.o IndexIVF.o IndexLSH.o IndexPQ.o \
IndexIVFPQ.o \
Clustering.o Heap.o VectorTransform.o index_io.o \
PolysemousTraining.o MetaIndexes.o Index.o \
ProductQuantizer.o AutoTune.o AuxIndexStructures.o \
IndexScalarQuantizer.o FaissException.o IndexHNSW.o \
IndexIVFFlat.o OnDiskInvertedLists.o
############################
# Installing
libfaiss.a: $(LIBOBJ)
ar r libfaiss.a $^
install: libfaiss.a libfaiss.$(SHAREDEXT) installdirs
cp libfaiss.a libfaiss.$(SHAREDEXT) $(DESTDIR)$(libdir)
cp *.h $(DESTDIR)$(includedir)/faiss/
libfaiss.$(SHAREDEXT): $(LIBOBJ)
$(CXX) $(LDFLAGS) $(FAISSSHAREDFLAGS) -o libfaiss.$(SHAREDEXT) $^ $(BLASLDFLAGS)
installdirs:
$(MKDIR_P) $(DESTDIR)$(libdir) $(DESTDIR)$(includedir)/faiss
.cpp.o:
$(CXX) $(CXXFLAGS) -c $< -o $@ $(FLAGS) $(EXTRAFLAGS)
uninstall:
rm $(DESTDIR)$(libdir)/libfaiss.a
rm $(DESTDIR)$(libdir)/libfaiss.$(SHAREDEXT)
rm -rf $(DESTDIR)$(includedir)/faiss
.c.o:
$(CC) $(CFLAGS) -c $< -o $@ $(FLAGS) $(EXTRAFLAGS)
utils.o: EXTRAFLAGS=$(BLASCFLAGS)
VectorTransform.o: EXTRAFLAGS=$(BLASCFLAGS)
ProductQuantizer.o: EXTRAFLAGS=$(BLASCFLAGS)
IndexHNSW.o: EXTRAFLAGS=$(BLASCFLAGS)
#############################
# Dependencies
# for MKL, the flags when generating a dynamic lib are different from
# the ones when making an executable, but by default they are the same
-include depend
BLASLDFLAGSSO ?= $(BLASLDFLAGS)
# The above makefile.dep is generated by the following target:
depend:
for i in $(SRC); do \
$(CXXCPP) $(CPPFLAGS) -MM $$i; \
done > depend
#############################
# pure C++ test in the test directory
tests/test_blas: tests/test_blas.cpp
$(CXX) $(CXXFLAGS) $< -o $@ $(LDFLAGS) $(BLASLDFLAGS) $(BLASCFLAGS)
# Tests
demos/demo_ivfpq_indexing: demos/demo_ivfpq_indexing.cpp libfaiss.a
$(CXX) -o $@ $(CXXFLAGS) $< libfaiss.a $(LDFLAGS) $(BLASLDFLAGS)
demos/demo_sift1M: demos/demo_sift1M.cpp libfaiss.a
$(CXX) -o $@ $(CXXFLAGS) $< libfaiss.a $(LDFLAGS) $(BLASLDFLAGS)
test: libfaiss.a
make -C tests run
#############################
# SWIG interfaces
HFILES = IndexFlat.h Index.h IndexLSH.h IndexPQ.h IndexIVF.h \
IndexIVFPQ.h VectorTransform.h index_io.h utils.h \
PolysemousTraining.h Heap.h MetaIndexes.h AuxIndexStructures.h \
Clustering.h hamming.h AutoTune.h IndexScalarQuantizer.h FaissException.h
# Demos
# also silently generates python/swigfaiss.py
python/swigfaiss_wrap.cxx: swigfaiss.swig $(HFILES)
$(SWIGEXEC) -python -c++ -Doverride= -o $@ $<
demos: libfaiss.a
make -C demos
# extension is .so even on the mac
python/_swigfaiss.so: python/swigfaiss_wrap.cxx libfaiss.a
$(CXX) -I. $(CXXFLAGS) $(LDFLAGS) $(PYTHONCFLAGS) $(SHAREDFLAGS) \
-o $@ $^ $(BLASLDFLAGSSO)
#############################
# Misc
_swigfaiss.so: python/_swigfaiss.so
cp python/_swigfaiss.so python/swigfaiss.py .
misc/test_blas: misc/test_blas.cpp
$(CXX) $(CXXFLAG) $(LDFLAGS) $(LIBS) -o $@ $^
#############################
# Dependencies.
# make dep > x
# then copy/paste from x by hand below
dep:
for i in $(patsubst %.o,%.cpp,$(LIBOBJ)) ; do \
cpp -MM -std=gnu++0x $$i ; \
done
hamming.o: hamming.cpp hamming.h Heap.h FaissAssert.h FaissException.h
utils.o: utils.cpp utils.h Heap.h AuxIndexStructures.h Index.h \
FaissAssert.h FaissException.h
IndexFlat.o: IndexFlat.cpp IndexFlat.h Index.h utils.h Heap.h \
FaissAssert.h FaissException.h AuxIndexStructures.h
IndexIVF.o: IndexIVF.cpp IndexIVF.h Index.h Clustering.h Heap.h utils.h \
hamming.h FaissAssert.h FaissException.h IndexFlat.h \
AuxIndexStructures.h
IndexLSH.o: IndexLSH.cpp IndexLSH.h Index.h VectorTransform.h utils.h \
Heap.h hamming.h FaissAssert.h FaissException.h
IndexPQ.o: IndexPQ.cpp IndexPQ.h Index.h ProductQuantizer.h Clustering.h \
Heap.h PolysemousTraining.h FaissAssert.h FaissException.h hamming.h
IndexIVFPQ.o: IndexIVFPQ.cpp IndexIVFPQ.h IndexIVF.h Index.h Clustering.h \
Heap.h IndexPQ.h ProductQuantizer.h PolysemousTraining.h utils.h \
IndexFlat.h hamming.h FaissAssert.h FaissException.h \
AuxIndexStructures.h
Clustering.o: Clustering.cpp Clustering.h Index.h utils.h Heap.h \
FaissAssert.h FaissException.h IndexFlat.h
Heap.o: Heap.cpp Heap.h
VectorTransform.o: VectorTransform.cpp VectorTransform.h Index.h utils.h \
Heap.h FaissAssert.h FaissException.h IndexPQ.h ProductQuantizer.h \
Clustering.h PolysemousTraining.h
index_io.o: index_io.cpp index_io.h FaissAssert.h FaissException.h \
IndexFlat.h Index.h VectorTransform.h IndexLSH.h IndexPQ.h \
ProductQuantizer.h Clustering.h Heap.h PolysemousTraining.h IndexIVF.h \
IndexIVFPQ.h IndexIVFFlat.h MetaIndexes.h IndexScalarQuantizer.h \
IndexHNSW.h utils.h OnDiskInvertedLists.h
PolysemousTraining.o: PolysemousTraining.cpp PolysemousTraining.h \
ProductQuantizer.h Clustering.h Index.h Heap.h utils.h hamming.h \
FaissAssert.h FaissException.h
MetaIndexes.o: MetaIndexes.cpp MetaIndexes.h Index.h FaissAssert.h \
FaissException.h Heap.h AuxIndexStructures.h
Index.o: Index.cpp IndexFlat.h Index.h FaissAssert.h FaissException.h
ProductQuantizer.o: ProductQuantizer.cpp ProductQuantizer.h Clustering.h \
Index.h Heap.h FaissAssert.h FaissException.h VectorTransform.h \
IndexFlat.h utils.h
AutoTune.o: AutoTune.cpp AutoTune.h Index.h FaissAssert.h \
FaissException.h utils.h Heap.h IndexFlat.h VectorTransform.h IndexLSH.h \
IndexPQ.h ProductQuantizer.h Clustering.h PolysemousTraining.h \
IndexIVF.h IndexIVFPQ.h IndexIVFFlat.h MetaIndexes.h \
IndexScalarQuantizer.h IndexHNSW.h
AuxIndexStructures.o: AuxIndexStructures.cpp AuxIndexStructures.h Index.h
IndexScalarQuantizer.o: IndexScalarQuantizer.cpp IndexScalarQuantizer.h \
IndexIVF.h Index.h Clustering.h Heap.h utils.h FaissAssert.h \
FaissException.h
FaissException.o: FaissException.cpp FaissException.h
IndexHNSW.o: IndexHNSW.cpp IndexHNSW.h IndexFlat.h Index.h IndexPQ.h \
ProductQuantizer.h Clustering.h Heap.h PolysemousTraining.h \
IndexScalarQuantizer.h IndexIVF.h utils.h FaissAssert.h FaissException.h \
IndexIVFPQ.h
IndexIVFFlat.o: IndexIVFFlat.cpp IndexIVFFlat.h IndexIVF.h Index.h \
Clustering.h Heap.h utils.h FaissAssert.h FaissException.h IndexFlat.h \
AuxIndexStructures.h
OnDiskInvertedLists.o: OnDiskInvertedLists.cpp OnDiskInvertedLists.h \
IndexIVF.h Index.h Clustering.h Heap.h FaissAssert.h FaissException.h
# Python
installdirs:
mkdir -p $(DESTDIR)$(PREFIX)/lib $(DESTDIR)$(PREFIX)/include/faiss
py: python/_swigfaiss.so
install: libfaiss.a libfaiss.$(SHAREDEXT) installdirs
cp libfaiss.a libfaiss.$(SHAREDEXT) $(DESTDIR)$(PREFIX)/lib/
cp ./*.h $(DESTDIR)$(PREFIX)/include/faiss/
python/_swigfaiss.so: libfaiss.a
$(MAKE) -C python _swigfaiss.so
uninstall:
rm $(DESTDIR)$(PREFIX)/lib/libfaiss.*
rm -rf $(DESTDIR)$(PREFIX)/include/faiss
clean:
rm -f libfaiss.a libfaiss.$(SHAREDEXT)* *.o \
lua/swigfaiss.so lua/swigfaiss_wrap.cxx \
python/_swigfaiss.so python/swigfaiss_wrap.cxx \
python/swigfaiss.py _swigfaiss.so swigfaiss.py
.env_ok:
ifeq ($(wildcard $(MAKEFILE_INC)),)
$(error Cannot find $(MAKEFILE_INC). Did you forget to copy the relevant file from ./example_makefiles?)
endif
ifeq ($(shell command -v $(CXX) 2>/dev/null),)
$(error Cannot find $(CXX), please refer to $(CURDIR)/makefile.inc to set up your environment)
endif
.swig_ok: .env_ok
ifeq ($(shell command -v $(SWIGEXEC) 2>/dev/null),)
$(error Cannot find $(SWIGEXEC), please refer to $(CURDIR)/makefile.inc to set up your environment)
endif
.PHONY: all clean default demos install installdirs py test uninstall
acinclude/ax_blas.m4 0 → 100644
View file @ cf18101f
# ===========================================================================
# https://www.gnu.org/software/autoconf-archive/ax_blas.html
# ===========================================================================
#
# SYNOPSIS
#
# AX_BLAS([ACTION-IF-FOUND[, ACTION-IF-NOT-FOUND]])
#
# DESCRIPTION
#
# This macro looks for a library that implements the BLAS linear-algebra
# interface (see http://www.netlib.org/blas/). On success, it sets the
# BLAS_LIBS output variable to hold the requisite library linkages.
#
# To link with BLAS, you should link with:
#
# $BLAS_LIBS $LIBS $FLIBS
#
# in that order. FLIBS is the output variable of the
# AC_F77_LIBRARY_LDFLAGS macro (called if necessary by AX_BLAS), and is
# sometimes necessary in order to link with F77 libraries. Users will also
# need to use AC_F77_DUMMY_MAIN (see the autoconf manual), for the same
# reason.
#
# Many libraries are searched for, from ATLAS to CXML to ESSL. The user
# may also use --with-blas=<lib> in order to use some specific BLAS
# library <lib>. In order to link successfully, however, be aware that you
# will probably need to use the same Fortran compiler (which can be set
# via the F77 env. var.) as was used to compile the BLAS library.
#
# ACTION-IF-FOUND is a list of shell commands to run if a BLAS library is
# found, and ACTION-IF-NOT-FOUND is a list of commands to run it if it is
# not found. If ACTION-IF-FOUND is not specified, the default action will
# define HAVE_BLAS.
#
# LICENSE
#
# Copyright (c) 2008 Steven G. Johnson <stevenj@alum.mit.edu>
#
# This program is free software: you can redistribute it and/or modify it
# under the terms of the GNU General Public License as published by the
# Free Software Foundation, either version 3 of the License, or (at your
# option) any later version.
#
# This program is distributed in the hope that it will be useful, but
# WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General
# Public License for more details.
#
# You should have received a copy of the GNU General Public License along
# with this program. If not, see <https://www.gnu.org/licenses/>.
#
# As a special exception, the respective Autoconf Macro's copyright owner
# gives unlimited permission to copy, distribute and modify the configure
# scripts that are the output of Autoconf when processing the Macro. You
# need not follow the terms of the GNU General Public License when using
# or distributing such scripts, even though portions of the text of the
# Macro appear in them. The GNU General Public License (GPL) does govern
# all other use of the material that constitutes the Autoconf Macro.
#
# This special exception to the GPL applies to versions of the Autoconf
# Macro released by the Autoconf Archive. When you make and distribute a
# modified version of the Autoconf Macro, you may extend this special
# exception to the GPL to apply to your modified version as well.
#serial 15
AU_ALIAS([ACX_BLAS], [AX_BLAS])
AC_DEFUN([AX_BLAS], [
AC_PREREQ(2.50)
# AC_REQUIRE([AC_F77_LIBRARY_LDFLAGS])
AC_REQUIRE([AC_CANONICAL_HOST])
ax_blas_ok=no
AC_ARG_WITH(blas,
[AS_HELP_STRING([--with-blas=<lib>], [use BLAS library <lib>])])
case $with_blas in
yes | "") ;;
no) ax_blas_ok=disable ;;
-* | */* | *.a | *.so | *.so.* | *.o) BLAS_LIBS="$with_blas" ;;
*) BLAS_LIBS="-l$with_blas" ;;
esac
# Get fortran linker names of BLAS functions to check for.
# AC_F77_FUNC(sgemm)
# AC_F77_FUNC(dgemm)
sgemm=sgemm_
dgemm=dgemm_
ax_blas_save_LIBS="$LIBS"
LIBS="$LIBS $FLIBS"
# First, check BLAS_LIBS environment variable
if test $ax_blas_ok = no; then
if test "x$BLAS_LIBS" != x; then
save_LIBS="$LIBS"; LIBS="$BLAS_LIBS $LIBS"
AC_MSG_CHECKING([for $sgemm in $BLAS_LIBS])
AC_TRY_LINK_FUNC($sgemm, [ax_blas_ok=yes], [BLAS_LIBS=""])
AC_MSG_RESULT($ax_blas_ok)
LIBS="$save_LIBS"
fi
fi
# BLAS linked to by default? (happens on some supercomputers)
if test $ax_blas_ok = no; then
save_LIBS="$LIBS"; LIBS="$LIBS"
AC_MSG_CHECKING([if $sgemm is being linked in already])
AC_TRY_LINK_FUNC($sgemm, [ax_blas_ok=yes])
AC_MSG_RESULT($ax_blas_ok)
LIBS="$save_LIBS"
fi
# BLAS in Intel MKL library?
if test $ax_blas_ok = no; then
# MKL for gfortran
if test x"$ac_cv_fc_compiler_gnu" = xyes; then
# 64 bit
if test $host_cpu = x86_64; then
AC_CHECK_LIB(mkl_gf_lp64, $sgemm,
[ax_blas_ok=yes;BLAS_LIBS="-lmkl_gf_lp64 -lmkl_sequential -lmkl_core -lpthread"],,
[-lmkl_gf_lp64 -lmkl_sequential -lmkl_core -lpthread])
# 32 bit
elif test $host_cpu = i686; then
AC_CHECK_LIB(mkl_gf, $sgemm,
[ax_blas_ok=yes;BLAS_LIBS="-lmkl_gf -lmkl_sequential -lmkl_core -lpthread"],,
[-lmkl_gf -lmkl_sequential -lmkl_core -lpthread])
fi
# MKL for other compilers (Intel, PGI, ...?)
else
# 64-bit
if test $host_cpu = x86_64; then
AC_CHECK_LIB(mkl_intel_lp64, $sgemm,
[ax_blas_ok=yes;BLAS_LIBS="-lmkl_intel_lp64 -lmkl_sequential -lmkl_core -lpthread"],,
[-lmkl_intel_lp64 -lmkl_sequential -lmkl_core -lpthread])
# 32-bit
elif test $host_cpu = i686; then
AC_CHECK_LIB(mkl_intel, $sgemm,
[ax_blas_ok=yes;BLAS_LIBS="-lmkl_intel -lmkl_sequential -lmkl_core -lpthread"],,
[-lmkl_intel -lmkl_sequential -lmkl_core -lpthread])
fi
fi
fi
# Old versions of MKL
if test $ax_blas_ok = no; then
AC_CHECK_LIB(mkl, $sgemm, [ax_blas_ok=yes;BLAS_LIBS="-lmkl -lguide -lpthread"],,[-lguide -lpthread])
fi
# BLAS in OpenBLAS library? (http://xianyi.github.com/OpenBLAS/)
if test $ax_blas_ok = no; then
AC_CHECK_LIB(openblas, $sgemm, [ax_blas_ok=yes
BLAS_LIBS="-lopenblas"])
fi
# BLAS in ATLAS library? (http://math-atlas.sourceforge.net/)
if test $ax_blas_ok = no; then
AC_CHECK_LIB(atlas, ATL_xerbla,
[AC_CHECK_LIB(f77blas, $sgemm,
[AC_CHECK_LIB(cblas, cblas_dgemm,
[ax_blas_ok=yes
BLAS_LIBS="-lcblas -lf77blas -latlas"],
[], [-lf77blas -latlas])],
[], [-latlas])])
fi
# BLAS in PhiPACK libraries? (requires generic BLAS lib, too)
if test $ax_blas_ok = no; then
AC_CHECK_LIB(blas, $sgemm,
[AC_CHECK_LIB(dgemm, $dgemm,
[AC_CHECK_LIB(sgemm, $sgemm,
[ax_blas_ok=yes; BLAS_LIBS="-lsgemm -ldgemm -lblas"],
[], [-lblas])],
[], [-lblas])])
fi
# BLAS in Apple vecLib library?
if test $ax_blas_ok = no; then
save_LIBS="$LIBS"; LIBS="-framework vecLib $LIBS"
AC_MSG_CHECKING([for $sgemm in -framework vecLib])
AC_TRY_LINK_FUNC($sgemm, [ax_blas_ok=yes;BLAS_LIBS="-framework vecLib"])
AC_MSG_RESULT($ax_blas_ok)
LIBS="$save_LIBS"
fi
# BLAS in Alpha CXML library?
if test $ax_blas_ok = no; then
AC_CHECK_LIB(cxml, $sgemm, [ax_blas_ok=yes;BLAS_LIBS="-lcxml"])
fi
# BLAS in Alpha DXML library? (now called CXML, see above)
if test $ax_blas_ok = no; then
AC_CHECK_LIB(dxml, $sgemm, [ax_blas_ok=yes;BLAS_LIBS="-ldxml"])
fi
# BLAS in Sun Performance library?
if test $ax_blas_ok = no; then
if test "x$GCC" != xyes; then # only works with Sun CC
AC_CHECK_LIB(sunmath, acosp,
[AC_CHECK_LIB(sunperf, $sgemm,
[BLAS_LIBS="-xlic_lib=sunperf -lsunmath"
ax_blas_ok=yes],[],[-lsunmath])])
fi
fi
# BLAS in SCSL library? (SGI/Cray Scientific Library)
if test $ax_blas_ok = no; then
AC_CHECK_LIB(scs, $sgemm, [ax_blas_ok=yes; BLAS_LIBS="-lscs"])
fi
# BLAS in SGIMATH library?
if test $ax_blas_ok = no; then
AC_CHECK_LIB(complib.sgimath, $sgemm,
[ax_blas_ok=yes; BLAS_LIBS="-lcomplib.sgimath"])
fi
# BLAS in IBM ESSL library? (requires generic BLAS lib, too)
if test $ax_blas_ok = no; then
AC_CHECK_LIB(blas, $sgemm,
[AC_CHECK_LIB(essl, $sgemm,
[ax_blas_ok=yes; BLAS_LIBS="-lessl -lblas"],
[], [-lblas $FLIBS])])
fi
# Generic BLAS library?
if test $ax_blas_ok = no; then
AC_CHECK_LIB(blas, $sgemm, [ax_blas_ok=yes; BLAS_LIBS="-lblas"])
fi
AC_SUBST(BLAS_LIBS)
LIBS="$ax_blas_save_LIBS"
# Finally, execute ACTION-IF-FOUND/ACTION-IF-NOT-FOUND:
if test x"$ax_blas_ok" = xyes; then
ifelse([$1],,AC_DEFINE(HAVE_BLAS,1,[Define if you have a BLAS library.]),[$1])
:
else
ax_blas_ok=no
$2
fi
])dnl AX_BLAS
acinclude/ax_cxx_compile_stdcxx.m4 0 → 100644
View file @ cf18101f
This diff is collapsed.
acinclude/ax_lapack.m4 0 → 100644
View file @ cf18101f
# ===========================================================================
# https://www.gnu.org/software/autoconf-archive/ax_lapack.html
# ===========================================================================
#
# SYNOPSIS
#
# AX_LAPACK([ACTION-IF-FOUND[, ACTION-IF-NOT-FOUND]])
#
# DESCRIPTION
#
# This macro looks for a library that implements the LAPACK linear-algebra
# interface (see http://www.netlib.org/lapack/). On success, it sets the
# LAPACK_LIBS output variable to hold the requisite library linkages.
#
# To link with LAPACK, you should link with:
#
# $LAPACK_LIBS $BLAS_LIBS $LIBS $FLIBS
#
# in that order. BLAS_LIBS is the output variable of the AX_BLAS macro,
# called automatically. FLIBS is the output variable of the
# AC_F77_LIBRARY_LDFLAGS macro (called if necessary by AX_BLAS), and is
# sometimes necessary in order to link with F77 libraries. Users will also
# need to use AC_F77_DUMMY_MAIN (see the autoconf manual), for the same
# reason.
#
# The user may also use --with-lapack=<lib> in order to use some specific
# LAPACK library <lib>. In order to link successfully, however, be aware
# that you will probably need to use the same Fortran compiler (which can
# be set via the F77 env. var.) as was used to compile the LAPACK and BLAS
# libraries.
#
# ACTION-IF-FOUND is a list of shell commands to run if a LAPACK library
# is found, and ACTION-IF-NOT-FOUND is a list of commands to run it if it
# is not found. If ACTION-IF-FOUND is not specified, the default action
# will define HAVE_LAPACK.
#
# LICENSE
#
# Copyright (c) 2009 Steven G. Johnson <stevenj@alum.mit.edu>
#
# This program is free software: you can redistribute it and/or modify it
# under the terms of the GNU General Public License as published by the
# Free Software Foundation, either version 3 of the License, or (at your
# option) any later version.
#
# This program is distributed in the hope that it will be useful, but
# WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General
# Public License for more details.
#
# You should have received a copy of the GNU General Public License along
# with this program. If not, see <https://www.gnu.org/licenses/>.
#
# As a special exception, the respective Autoconf Macro's copyright owner
# gives unlimited permission to copy, distribute and modify the configure
# scripts that are the output of Autoconf when processing the Macro. You
# need not follow the terms of the GNU General Public License when using
# or distributing such scripts, even though portions of the text of the
# Macro appear in them. The GNU General Public License (GPL) does govern
# all other use of the material that constitutes the Autoconf Macro.
#
# This special exception to the GPL applies to versions of the Autoconf
# Macro released by the Autoconf Archive. When you make and distribute a
# modified version of the Autoconf Macro, you may extend this special
# exception to the GPL to apply to your modified version as well.
#serial 8
AU_ALIAS([ACX_LAPACK], [AX_LAPACK])
AC_DEFUN([AX_LAPACK], [
AC_REQUIRE([AX_BLAS])
ax_lapack_ok=no
AC_ARG_WITH(lapack,
[AS_HELP_STRING([--with-lapack=<lib>], [use LAPACK library <lib>])])
case $with_lapack in
yes | "") ;;
no) ax_lapack_ok=disable ;;
-* | */* | *.a | *.so | *.so.* | *.o) LAPACK_LIBS="$with_lapack" ;;
*) LAPACK_LIBS="-l$with_lapack" ;;
esac
# Get fortran linker name of LAPACK function to check for.
# AC_F77_FUNC(cheev)
cheev=cheev_
# We cannot use LAPACK if BLAS is not found
if test "x$ax_blas_ok" != xyes; then
ax_lapack_ok=noblas
LAPACK_LIBS=""
fi
# First, check LAPACK_LIBS environment variable
if test "x$LAPACK_LIBS" != x; then
save_LIBS="$LIBS"; LIBS="$LAPACK_LIBS $BLAS_LIBS $LIBS $FLIBS"
AC_MSG_CHECKING([for $cheev in $LAPACK_LIBS])
AC_TRY_LINK_FUNC($cheev, [ax_lapack_ok=yes], [LAPACK_LIBS=""])
AC_MSG_RESULT($ax_lapack_ok)
LIBS="$save_LIBS"
if test $ax_lapack_ok = no; then
LAPACK_LIBS=""
fi
fi
# LAPACK linked to by default? (is sometimes included in BLAS lib)
if test $ax_lapack_ok = no; then
save_LIBS="$LIBS"; LIBS="$LIBS $BLAS_LIBS $FLIBS"
AC_CHECK_FUNC($cheev, [ax_lapack_ok=yes])
LIBS="$save_LIBS"
fi
# Generic LAPACK library?
for lapack in lapack lapack_rs6k; do
if test $ax_lapack_ok = no; then
save_LIBS="$LIBS"; LIBS="$BLAS_LIBS $LIBS"
AC_CHECK_LIB($lapack, $cheev,
[ax_lapack_ok=yes; LAPACK_LIBS="-l$lapack"], [], [$FLIBS])
LIBS="$save_LIBS"
fi
done
AC_SUBST(LAPACK_LIBS)
# Finally, execute ACTION-IF-FOUND/ACTION-IF-NOT-FOUND:
if test x"$ax_lapack_ok" = xyes; then
ifelse([$1],,AC_DEFINE(HAVE_LAPACK,1,[Define if you have LAPACK library.]),[$1])
:
else
ax_lapack_ok=no
$2
fi
])dnl AX_LAPACK
acinclude/fa_check_cuda.m4 0 → 100644
View file @ cf18101f
AC_DEFUN([FA_CHECK_CUDA], [
AC_ARG_WITH(cuda,
[AS_HELP_STRING([--with-cuda=<prefix>], [prefix of the CUDA installation])])
case $with_cuda in
"") cuda_prefix=/usr/local/cuda ;;
*) cuda_prefix="$with_cuda"
esac
AC_CHECK_PROG(NVCC, "nvcc", "$cuda_prefix/bin/nvcc", "", "$cuda_prefix/bin")
fa_nvcc_bin=$NVCC
if test x$fa_nvcc_bin != x; then
fa_save_CPPFLAGS="$CPPFLAGS"
fa_save_LDFLAGS="$LDFLAGS"
fa_save_LIBS="$LIBS"
NVCC_CPPFLAGS="-I$cuda_prefix/include"
NVCC_LDFLAGS="-L$cuda_prefix/lib64"
CPPFLAGS="$NVCC_CPPFLAGS $CPPFLAGS"
LDFLAGS="$NVCC_LDFLAGS $LDFLAGS"
AC_CHECK_HEADER([cuda.h], [], AC_MSG_FAILURE([Couldn't find cuda.h]))
AC_CHECK_LIB([cuda], [cuInit], [], AC_MSG_FAILURE([Couldn't find libcuda]))
AC_CHECK_LIB([cublas], [cublasAlloc], [], AC_MSG_FAILURE([Couldn't find libcublas]))
AC_CHECK_LIB([cudart], [cudaSetDevice], [], AC_MSG_FAILURE([Couldn't find libcudart]))
NVCC_LIBS="$LIBS"
NVCC_CPPFLAGS="$CPPFLAGS"
NVCC_LDFLAGS="$LDFLAGS"
CPPFLAGS="$fa_save_CPPFLAGS"
LDFLAGS="$fa_save_LDFLAGS"
LIBS="$fa_save_LIBS"
else
cuda_prefix=""
fi
AC_SUBST(NVCC)
AC_SUBST(NVCC_CPPFLAGS)
AC_SUBST(NVCC_LDFLAGS)
AC_SUBST(NVCC_LIBS)
AC_SUBST(CUDA_PREFIX, $cuda_prefix)
])
acinclude/fa_numpy.m4 0 → 100644
View file @ cf18101f
AC_DEFUN([FA_NUMPY], [
AC_REQUIRE([FA_PYTHON])
AC_MSG_CHECKING([for numpy headers path])
fa_numpy_headers=`$PYTHON -c "import numpy; print(numpy.get_include())"`
if test $? == 0; then
if test x$fa_numpy_headers != x; then
AC_MSG_RESULT($fa_numpy_headers)
AC_SUBST(NUMPY_INCLUDE, $fa_numpy_headers)
else
AC_MSG_RESULT([not found])
AC_MSG_WARN([You won't be able to build the python interface.])
fi
else
AC_MSG_RESULT([not found])
AC_MSG_WARN([You won't be able to build the python interface.])
fi
])dnl
acinclude/fa_prog_nm.m4 0 → 100644
View file @ cf18101f
dnl
dnl Check for an nm(1) utility.
dnl
AC_DEFUN([FA_PROG_NM],
[
case "${NM-unset}" in
unset) AC_CHECK_PROGS(NM, nm, nm) ;;
*) AC_CHECK_PROGS(NM, $NM nm, nm) ;;
esac
AC_MSG_CHECKING(nm flags)
case "${NMFLAGS-unset}" in
unset) NMFLAGS= ;;
esac
AC_MSG_RESULT($NMFLAGS)
AC_SUBST(NMFLAGS)
])
acinclude/fa_prog_swig.m4 0 → 100644
View file @ cf18101f
AC_DEFUN([FA_PROG_SWIG], [
AC_ARG_WITH(swig,
[AS_HELP_STRING([--with-swig=<bin>], [use SWIG binary <bin>])])
case $with_swig in
"") SWIG_BIN=swig ;;
*) SWIG_BIN="$with_swig"
esac
AC_CHECK_PROG(SWIG, $SWIG_BIN, $SWIG_BIN)
AC_SUBST(SWIG)
])
acinclude/fa_python.m4 0 → 100644
View file @ cf18101f
AC_DEFUN([FA_PYTHON], [
AC_ARG_WITH(python,
[AS_HELP_STRING([--with-python=<bin>], [use Python binary <bin>])])
case $with_python in
"") PYTHON_BIN=python ;;
*) PYTHON_BIN="$with_python"
esac
AC_CHECK_PROG(PYTHON, $PYTHON_BIN, $PYTHON_BIN)
fa_python_bin=$PYTHON
AC_ARG_WITH(python-config,
[AS_HELP_STRING([--with-python-config=<bin>], [use Python config binary <bin>])])
case $with_python_config in
"") PYTHON_CFG_BIN="${PYTHON_BIN}-config" ;;
*) PYTHON_CFG_BIN="$with_python_config"
esac
AC_CHECK_PROG(PYTHON_CFG, $PYTHON_CFG_BIN, $PYTHON_CFG_BIN)
fa_python_cfg_bin=$PYTHON_CFG
if test x$fa_python_cfg_bin != x; then
AC_MSG_CHECKING([for Python C flags])
fa_python_cflags=`${PYTHON_CFG} --includes`
if test x"$fa_python_cflags" == x; then
AC_MSG_RESULT([not found])
AC_MSG_WARN([You won't be able to build the python interface.])
else
AC_MSG_RESULT($fa_python_cflags)
AC_SUBST(PYTHON_CFLAGS, $fa_python_cflags)
fi
AC_MSG_CHECKING([for Python ld flags])
fa_python_ldflags=`${PYTHON_CFG} --ldflags`
if test x"$fa_python_ldflags" == x; then
AC_MSG_RESULT([not found])
else
AC_MSG_RESULT($fa_python_ldflags)
AC_SUBST(PYTHON_LDFLAGS, $fa_python_ldflags)
fi
else
AC_MSG_WARN([You won't be able to build the python interface.])
fi
])
build-aux/config.guess 0 → 100755
View file @ cf18101f
This diff is collapsed.
build-aux/config.sub 0 → 100755
View file @ cf18101f
This diff is collapsed.
build-aux/install-sh 0 → 100755
View file @ cf18101f
This diff is collapsed.
configure 0 → 100755
View file @ cf18101f
This diff is collapsed.
configure.ac 0 → 100644
View file @ cf18101f
# -*- Autoconf -*-
# Process this file with autoconf to produce a configure script.
AC_PREREQ([2.69])
AC_INIT([faiss], [1.0])
AC_COPYRIGHT([Copyright (c) 2015-present, Facebook, Inc.
All rights reserved.
This source code is licensed under the BSD+Patents license found in the
LICENSE file in the root directory of this source tree.])
AC_CONFIG_SRCDIR([Index.h])
AC_CONFIG_AUX_DIR([build-aux])
AC_CONFIG_MACRO_DIR([acinclude])
: ${CXXFLAGS="-g -O3 -Wall -Wextra"}
# Checks for programs.
AC_LANG(C++)
AC_PROG_CXX
AX_CXX_COMPILE_STDCXX([11], [noext], [mandatory])
AC_PROG_CPP
AC_PROG_MAKE_SET
AC_PROG_MKDIR_P
FA_PYTHON
if test x$PYTHON != x; then
FA_NUMPY
fi
FA_PROG_SWIG
FA_CHECK_CUDA
# Checks for header files.
AC_CHECK_HEADERS([float.h limits.h stddef.h stdint.h stdlib.h string.h sys/time.h unistd.h])
# Checks for typedefs, structures, and compiler characteristics.
AC_CHECK_HEADER_STDBOOL
AC_C_INLINE
AC_TYPE_INT32_T
AC_TYPE_INT64_T
AC_C_RESTRICT
AC_TYPE_SIZE_T
AC_TYPE_UINT16_T
AC_TYPE_UINT32_T
AC_TYPE_UINT64_T
AC_TYPE_UINT8_T
# Checks for library functions.
AC_FUNC_MALLOC
AC_FUNC_MMAP
AC_CHECK_FUNCS([clock_gettime floor gettimeofday memmove memset munmap pow sqrt strerror strstr])
AC_OPENMP
AX_BLAS
if test "x$ax_blas_ok" == "xno"; then
AC_MSG_ERROR([An implementation of BLAS is required but none was found.])
fi
AX_LAPACK
if test "x$ax_lapack_ok" == "xno"; then
AC_MSG_ERROR([An implementation of LAPACK is required but none was found.])
fi
AC_CONFIG_FILES([makefile.inc])
AC_OUTPUT
demos/Makefile 0 → 100644
View file @ cf18101f
# Copyright (c) 2015-present, Facebook, Inc.
# All rights reserved.
#
# This source code is licensed under the BSD+Patents license found in the
# LICENSE file in the root directory of this source tree.
-include ../makefile.inc
DEMOS_SRC=$(wildcard demo_*.cpp)
DEMOS=$(DEMOS_SRC:.cpp=)
all: $(DEMOS)
clean:
rm -f $(DEMOS)
%: %.cpp ../libfaiss.a
$(CXX) -o $@ $(CXXFLAGS) $^ $(LDFLAGS) $(LIBS)
.PHONY: all clean
depend 0 → 100644
View file @ cf18101f
AutoTune.o: AutoTune.cpp AutoTune.h Index.h FaissAssert.h \
FaissException.h utils.h Heap.h IndexFlat.h VectorTransform.h IndexLSH.h \
IndexPQ.h ProductQuantizer.h Clustering.h PolysemousTraining.h \
IndexIVF.h IndexIVFPQ.h IndexIVFFlat.h MetaIndexes.h \
IndexScalarQuantizer.h IndexHNSW.h
AuxIndexStructures.o: AuxIndexStructures.cpp AuxIndexStructures.h Index.h
Clustering.o: Clustering.cpp Clustering.h Index.h utils.h Heap.h \
FaissAssert.h FaissException.h IndexFlat.h
FaissException.o: FaissException.cpp FaissException.h
Heap.o: Heap.cpp Heap.h
Index.o: Index.cpp IndexFlat.h Index.h FaissAssert.h FaissException.h
IndexFlat.o: IndexFlat.cpp IndexFlat.h Index.h utils.h Heap.h \
FaissAssert.h FaissException.h AuxIndexStructures.h
IndexHNSW.o: IndexHNSW.cpp IndexHNSW.h IndexFlat.h Index.h IndexPQ.h \
ProductQuantizer.h Clustering.h Heap.h PolysemousTraining.h \
IndexScalarQuantizer.h IndexIVF.h utils.h FaissAssert.h FaissException.h \
IndexIVFPQ.h
IndexIVF.o: IndexIVF.cpp IndexIVF.h Index.h Clustering.h Heap.h utils.h \
hamming.h FaissAssert.h FaissException.h IndexFlat.h \
AuxIndexStructures.h
IndexIVFFlat.o: IndexIVFFlat.cpp IndexIVFFlat.h IndexIVF.h Index.h \
Clustering.h Heap.h utils.h FaissAssert.h FaissException.h IndexFlat.h \
AuxIndexStructures.h
IndexIVFPQ.o: IndexIVFPQ.cpp IndexIVFPQ.h IndexIVF.h Index.h Clustering.h \
Heap.h IndexPQ.h ProductQuantizer.h PolysemousTraining.h utils.h \
IndexFlat.h hamming.h FaissAssert.h FaissException.h \
AuxIndexStructures.h
IndexLSH.o: IndexLSH.cpp IndexLSH.h Index.h VectorTransform.h utils.h \
Heap.h hamming.h FaissAssert.h FaissException.h
IndexPQ.o: IndexPQ.cpp IndexPQ.h Index.h ProductQuantizer.h Clustering.h \
Heap.h PolysemousTraining.h FaissAssert.h FaissException.h \
AuxIndexStructures.h hamming.h
IndexScalarQuantizer.o: IndexScalarQuantizer.cpp IndexScalarQuantizer.h \
IndexIVF.h Index.h Clustering.h Heap.h utils.h FaissAssert.h \
FaissException.h
MetaIndexes.o: MetaIndexes.cpp MetaIndexes.h Index.h FaissAssert.h \
FaissException.h Heap.h AuxIndexStructures.h
OnDiskInvertedLists.o: OnDiskInvertedLists.cpp OnDiskInvertedLists.h \
IndexIVF.h Index.h Clustering.h Heap.h FaissAssert.h FaissException.h
PolysemousTraining.o: PolysemousTraining.cpp PolysemousTraining.h \
ProductQuantizer.h Clustering.h Index.h Heap.h utils.h hamming.h \
FaissAssert.h FaissException.h
ProductQuantizer.o: ProductQuantizer.cpp ProductQuantizer.h Clustering.h \
Index.h Heap.h FaissAssert.h FaissException.h VectorTransform.h \
IndexFlat.h utils.h
VectorTransform.o: VectorTransform.cpp VectorTransform.h Index.h utils.h \
Heap.h FaissAssert.h FaissException.h IndexPQ.h ProductQuantizer.h \
Clustering.h PolysemousTraining.h
hamming.o: hamming.cpp hamming.h Heap.h FaissAssert.h FaissException.h
index_io.o: index_io.cpp index_io.h FaissAssert.h FaissException.h \
IndexFlat.h Index.h VectorTransform.h IndexLSH.h IndexPQ.h \
ProductQuantizer.h Clustering.h Heap.h PolysemousTraining.h IndexIVF.h \
IndexIVFPQ.h IndexIVFFlat.h MetaIndexes.h IndexScalarQuantizer.h \
IndexHNSW.h utils.h OnDiskInvertedLists.h
utils.o: utils.cpp utils.h Heap.h AuxIndexStructures.h Index.h \
FaissAssert.h FaissException.h
example_makefiles/makefile.inc.Linux
View file @ cf18101f
# -*- makefile -*-
# Copyright (c) 2015-present, Facebook, Inc.
# All rights reserved.
#
# This source code is licensed under the BSD+Patents license found in the
# LICENSE file in the root directory of this source tree.
# -*- makefile -*-
# tested on CentOS 7, Ubuntu 16 and Ubuntu 14, see below to adjust flags to distribution.
CC=gcc
CXX=g++
CFLAGS=-fPIC -m64 -Wall -g -O3 -mavx -msse4 -mpopcnt -fopenmp -Wno-sign-compare
CXXFLAGS=$(CFLAGS) -std=c++11
LDFLAGS=-g -fPIC -fopenmp
CXX = g++ -std=c++11
CXXFLAGS = -fPIC -m64 -Wall -g -O3 -fopenmp -Wno-sign-compare
CPUFLAGS = -mavx -msse4 -mpopcnt
LDFLAGS = -fPIC -fopenmp
# common linux flags
SHAREDEXT=so
SHAREDFLAGS=-shared
FAISSSHAREDFLAGS=-shared
SHAREDEXT = so
SHAREDFLAGS = -shared
MKDIR_P = mkdir -p
prefix ?= /usr/local
exec_prefix ?= ${prefix}
libdir = ${exec_prefix}/lib
includedir = ${prefix}/include
##########################################################################
# Uncomment one of the 4 BLAS/Lapack implementation options
......@@ -37,47 +38,44 @@ FAISSSHAREDFLAGS=-shared
#
# https://software.intel.com/en-us/articles/intel-mkl-link-line-advisor
#
# The latest tested version is MLK 2017.0.098 (2017 Initial Release) and can
# The latest tested version is MKL 2017.0.098 (2017 Initial Release) and can
# be downloaded here:
#
# https://registrationcenter.intel.com/en/forms/?productid=2558&licensetype=2
#
# The following settings are working if MLK is installed on its default folder:
# The following settings are working if MKL is installed on its default folder:
#
# MKLROOT = /opt/intel/compilers_and_libraries/linux/mkl/
#
# MKLROOT=/opt/intel/compilers_and_libraries/linux/mkl/
# LDFLAGS += -Wl,--no-as-needed -L$(MKLROOT)/lib/intel64
# LIBS += -lmkl_intel_ilp64 -lmkl_core -lmkl_gnu_thread -ldl -lpthread
#
# BLASLDFLAGS=-Wl,--no-as-needed -L$(MKLROOT)/lib/intel64 -lmkl_intel_ilp64 \
# -lmkl_core -lmkl_gnu_thread -ldl -lpthread
# CPPFLAGS += -DFINTEGER=long
#
# BLASCFLAGS=-DFINTEGER=long
# You may have to set the LD_LIBRARY_PATH=$MKLROOT/lib/intel64 at runtime.
#
# you may have to set the LD_LIBRARY_PATH=$MKLROOT/lib/intel64 at runtime.
# If at runtime you get the error:
# Intel MKL FATAL ERROR: Cannot load libmkl_avx2.so or libmkl_def.so.
# You may add set
# Intel MKL FATAL ERROR: Cannot load libmkl_avx2.so or libmkl_def.so
# you may set
# LD_PRELOAD=$MKLROOT/lib/intel64/libmkl_core.so:$MKLROOT/lib/intel64/libmkl_sequential.so
# at runtime as well.
#
# 2. Openblas
#
# The library contains both BLAS and Lapack. About 30% slower than MKL. Please see
# The library contains both BLAS and Lapack. About 30% slower than MKL. Please see
# https://github.com/facebookresearch/faiss/wiki/Troubleshooting#slow-brute-force-search-with-openblas
# to fix performance problemes with OpenBLAS
BLASCFLAGS=-DFINTEGER=int
# This is for Centos:
BLASLDFLAGS?=/usr/lib64/libopenblas.so.0
# to fix performance problemes with OpenBLAS
# for Ubuntu 16:
# sudo apt-get install libopenblas-dev python-numpy python-dev
# BLASLDFLAGS?=/usr/lib/libopenblas.so.0
# for Ubuntu 14:
# sudo apt-get install libopenblas-dev liblapack3 python-numpy python-dev
# BLASLDFLAGS?=/usr/lib/libopenblas.so.0 /usr/lib/lapack/liblapack.so.3.0
#
CPPFLAGS += -DFINTEGER=int
LIBS += -lopenblas -llapack
# 3. Atlas
#
# Automatically tuned linear algebra package. As the name indicates,
......@@ -93,8 +91,8 @@ BLASLDFLAGS?=/usr/lib64/libopenblas.so.0
# This is just a compiled version of the reference BLAS
# implementation, that is not optimized at all.
#
# BLASCFLAGS=-DFINTEGER=int
# BLASLDFLAGS=/usr/lib64/libblas.so.3 /usr/lib64/liblapack.so.3.2
# CPPFLAGS += -DFINTEGER=int
# LIBS += /usr/lib64/libblas.so.3 /usr/lib64/liblapack.so.3.2
#
......@@ -103,7 +101,7 @@ BLASLDFLAGS?=/usr/lib64/libopenblas.so.0
##########################################################################
# SWIG executable. This should be at least version 3.x
SWIGEXEC=swig
SWIG = swig
# The Python include directories for a given python executable can
# typically be found with
......@@ -117,8 +115,8 @@ SWIGEXEC=swig
# python3 -c "import numpy ; print(numpy.get_include())"
#
PYTHONCFLAGS=-I/usr/include/python2.7/ -I/usr/lib64/python2.7/site-packages/numpy/core/include/
PYTHONCFLAGS = -I/usr/include/python2.7/ -I/usr/lib64/python2.7/site-packages/numpy/core/include/
PYTHONLIB = -lpython
###########################################################################
# Cuda GPU flags
......@@ -127,26 +125,16 @@ PYTHONCFLAGS=-I/usr/include/python2.7/ -I/usr/lib64/python2.7/site-packages/nump
# root of the cuda 8 installation
CUDAROOT=/usr/local/cuda-8.0/
CUDACFLAGS=-I$(CUDAROOT)/include
NVCC=$(CUDAROOT)/bin/nvcc
NVCCFLAGS= $(CUDAFLAGS) \
-I $(CUDAROOT)/targets/x86_64-linux/include/ \
-Xcompiler -fPIC \
-Xcudafe --diag_suppress=unrecognized_attribute \
-gencode arch=compute_35,code="compute_35" \
-gencode arch=compute_52,code="compute_52" \
-gencode arch=compute_60,code="compute_60" \
--std c++11 -lineinfo \
-ccbin $(CXX) -DFAISS_USE_FLOAT16
# BLAS LD flags for nvcc (used to generate an executable)
# if BLASLDFLAGS contains several flags, each one may
# need to be prepended with -Xlinker
BLASLDFLAGSNVCC=-Xlinker $(BLASLDFLAGS)
# Same, but to generate a .so
BLASLDFLAGSSONVCC=-Xlinker $(BLASLDFLAGS)
CUDAROOT = /usr/local/cuda-8.0
NVCC = $(CUDAROOT)/bin/nvcc
NVCCLDFLAGS = -L$(CUDAROOT)/lib64
NVCCLIBS = -lcudart -lcublas -lcuda
CUDACFLAGS = -I$(CUDAROOT)/include
NVCCFLAGS = -I $(CUDAROOT)/targets/x86_64-linux/include/ \
-Xcompiler -fPIC \
-Xcudafe --diag_suppress=unrecognized_attribute \
-gencode arch=compute_35,code="compute_35" \
-gencode arch=compute_52,code="compute_52" \
-gencode arch=compute_60,code="compute_60" \
-lineinfo \
-ccbin $(CXX) -DFAISS_USE_FLOAT16
example_makefiles/makefile.inc.Mac.brew
View file @ cf18101f
# -*- makefile -*-
# Copyright (c) 2015-present, Facebook, Inc.
# All rights reserved.
#
# This source code is licensed under the BSD+Patents license found in the
# LICENSE file in the root directory of this source tree.
# Tested on macOS Sierra (10.12.2) with llvm installed using Homebrew (https://brew.sh)
# brew install llvm
CXX=/usr/local/opt/llvm/bin/clang++
CFLAGS=-fPIC -m64 -Wall -g -O3 -msse4 -mpopcnt -fopenmp -Wno-sign-compare -I/usr/local/opt/llvm/include
CXXFLAGS=$(CFLAGS) -std=c++11
CXX = /usr/local/opt/llvm/bin/clang++ -std=c++11
CXXFLAGS = -fPIC -m64 -Wall -g -O3 -fopenmp -Wno-sign-compare -I/usr/local/opt/llvm/include
CPUFLAGS = -msse4 -mpopcnt
LLVM_VERSION_PATH=$(shell ls -rt /usr/local/Cellar/llvm/ | tail -n1)
LDFLAGS=-g -fPIC -fopenmp -L/usr/local/opt/llvm/lib -L/usr/local/Cellar/llvm/${LLVM_VERSION_PATH}/lib
LDFLAGS = -fPIC -fopenmp -L/usr/local/opt/llvm/lib -L/usr/local/Cellar/llvm/${LLVM_VERSION_PATH}/lib
# common mac flags
SHAREDEXT=dylib
SHAREDFLAGS=-Wl,-F. -bundle -undefined dynamic_lookup
FAISSSHAREDFLAGS=-dynamiclib
SHAREDEXT = dylib
SHAREDFLAGS = -dynamiclib
MKDIR_P = mkdir -p
# wrapldflags=""
# sharedext=dylib
# sharedflags="-dynamiclib"
# yaelsharedflags="$sharedflags -install_name $yaelprefix/yael/libyael.dylib"
prefix ?= /usr/local
exec_prefix ?= ${prefix}
libdir = ${exec_prefix}/lib
includedir = ${prefix}/include
##########################################################################
# Uncomment one of the 4 BLAS/Lapack implementation options
......@@ -34,28 +39,29 @@ FAISSSHAREDFLAGS=-dynamiclib
#
# https://software.intel.com/en-us/articles/intel-mkl-link-line-advisor
#
# for a start on setting the link flags. On version IntelComposerXE
# 2015.0.090, the following flags work
# The latest tested version is MKL 2017.0.098 (2017 Initial Release) and can
# be downloaded here:
#
# MKLROOT=$(HOME)/fbsource/fbcode/third-party2//IntelComposerXE/2015.0.090/gcc-4.8.1-glibc-2.17/c3f970a/mkl
# https://registrationcenter.intel.com/en/forms/?productid=2558&licensetype=2
#
# BLASLDFLAGS=-Wl,--no-as-needed -L$(MKLROOT)/lib/intel64 -lmkl_intel_ilp64 \
# -lmkl_core -lmkl_gnu_thread -ldl -lpthread
# The following settings are working if MKL is installed on its default folder:
#
# the ilp64 means that the integers are 64-bit.
# MKLROOT = /opt/intel/compilers_and_libraries/linux/mkl/
#
# BLASLDFLAGS=-DFINTEGER=long
# LDFLAGS += -Wl,--no-as-needed -L$(MKLROOT)/lib/intel64
# LIBS += -lmkl_intel_ilp64 -lmkl_core -lmkl_gnu_thread -ldl -lpthread
#
# you may have to set the LD_LIBRARY_PATH=$MKLROOT/lib/intel64 at runtime
# CPPFLAGS += -DFINTEGER=long
#
# You may have to set the LD_LIBRARY_PATH=$MKLROOT/lib/intel64 at runtime.
#
# 2. Openblas
#
# The library contains both BLAS and Lapack. Install with brew install OpenBLAS
#
# BLASCFLAGS=-DFINTEGER=int
# BLASLDFLAGS=/usr/local/opt/openblas/lib/libblas.dylib
# CPPFLAGS += -DFINTEGER=int
# LIBS += /usr/local/opt/openblas/lib/libblas.dylib
#
#
......@@ -65,8 +71,8 @@ FAISSSHAREDFLAGS=-dynamiclib
# as it is provided by default on the mac. It is not very fast, though.
#
BLASCFLAGS=-DFINTEGER=int
BLASLDFLAGS=-framework Accelerate
CPPFLAGS += -DFINTEGER=int
LIBS += -framework Accelerate
......@@ -77,13 +83,14 @@ BLASLDFLAGS=-framework Accelerate
# SWIG executable. This should be at least version 3.x
# brew install swig
SWIGEXEC=/usr/local/bin/swig
SWIG = /usr/local/bin/swig
# The Python include directories for the current python executable
PYTHON_INC=$(shell python -c "import distutils.sysconfig; print(distutils.sysconfig.get_python_inc())")
NUMPY_INC=$(shell python -c "import numpy ; print(numpy.get_include())")
PYTHONCFLAGS=-I${PYTHON_INC} -I${NUMPY_INC}
PYTHONLIB=-lpython
##########################################################################
# Faiss GPU
......
example_makefiles/makefile.inc.Mac.port
View file @ cf18101f
# -*- makefile -*-
# Copyright (c) 2015-present, Facebook, Inc.
# All rights reserved.
#
# This source code is licensed under the BSD+Patents license found in the
# LICENSE file in the root directory of this source tree.
#
# tested on Mac OS X 10.12.2 Sierra with additional software installed via macports
# The system default clang does not support openmp
# You can install an openmp compatible g++ with macports:
# port install g++-mp-6
CXX=/opt/local/bin/g++-mp-6
CFLAGS=-fPIC -m64 -Wall -g -O3 -msse4 -mpopcnt -fopenmp -Wno-sign-compare
CXXFLAGS=$(CFLAGS) -std=c++11
LDFLAGS=-g -fPIC -fopenmp
CXX = /opt/local/bin/g++-mp-6 -std=c++11
CXXFLAGS = -fPIC -m64 -Wall -g -O3 -fopenmp -Wno-sign-compare
CPUFLAGS = -msse4 -mpopcnt
LDFLAGS = -g -fPIC -fopenmp
# common linux flags
SHAREDEXT=dylib
SHAREDFLAGS=-Wl,-F. -bundle -undefined dynamic_lookup
FAISSSHAREDFLAGS=-dynamiclib
SHAREDEXT = dylib
SHAREDFLAGS = -dynamiclib
MKDIR_P = mkdir -p
# wrapldflags=""
# sharedext=dylib
# sharedflags="-dynamiclib"
# yaelsharedflags="$sharedflags -install_name $yaelprefix/yael/libyael.dylib"
prefix ?= /usr/local
exec_prefix ?= ${prefix}
libdir = ${exec_prefix}/lib
includedir = ${prefix}/include
##########################################################################
# Uncomment one of the 4 BLAS/Lapack implementation options
......@@ -39,28 +41,29 @@ FAISSSHAREDFLAGS=-dynamiclib
#
# https://software.intel.com/en-us/articles/intel-mkl-link-line-advisor
#
# for a start on setting the link flags. On version IntelComposerXE
# 2015.0.090, the following flags work
# The latest tested version is MKL 2017.0.098 (2017 Initial Release) and can
# be downloaded here:
#
# MKLROOT=$(HOME)/fbsource/fbcode/third-party2//IntelComposerXE/2015.0.090/gcc-4.8.1-glibc-2.17/c3f970a/mkl
# https://registrationcenter.intel.com/en/forms/?productid=2558&licensetype=2
#
# BLASLDFLAGS=-Wl,--no-as-needed -L$(MKLROOT)/lib/intel64 -lmkl_intel_ilp64 \
# -lmkl_core -lmkl_gnu_thread -ldl -lpthread
# The following settings are working if MKL is installed on its default folder:
#
# the ilp64 means that the integers are 64-bit.
# MKLROOT = /opt/intel/compilers_and_libraries/linux/mkl/
#
# BLASLDFLAGS=-DFINTEGER=long
# LDFLAGS += -Wl,--no-as-needed -L$(MKLROOT)/lib/intel64
# LIBS += -lmkl_intel_ilp64 -lmkl_core -lmkl_gnu_thread -ldl -lpthread
#
# you may have to set the LD_LIBRARY_PATH=$MKLROOT/lib/intel64 at runtime
# CPPFLAGS += -DFINTEGER=long
#
# You may have to set the LD_LIBRARY_PATH=$MKLROOT/lib/intel64 at runtime.
#
# 2. Openblas
#
# The library contains both BLAS and Lapack. Install with port install OpenBLAS
#
# BLASCFLAGS=-DFINTEGER=int
# BLASLDFLAGS=/opt/local/lib/libopenblas.dylib
# CPPFLAGS += -DFINTEGER=int
# LIBS += /opt/local/lib/libopenblas.dylib
#
#
......@@ -70,8 +73,8 @@ FAISSSHAREDFLAGS=-dynamiclib
# as it is provided by default on the mac. It is not very fast, though.
#
BLASCFLAGS=-DFINTEGER=int
BLASLDFLAGS=-framework Accelerate
CPPFLAGS += -DFINTEGER=int
LIBS += -framework Accelerate
......@@ -82,7 +85,7 @@ BLASLDFLAGS=-framework Accelerate
# SWIG executable. This should be at least version 3.x
# port install swig swig-python
SWIGEXEC=/opt/local/bin/swig
SWIG = /opt/local/bin/swig
# The Python include directories for the current python executable can
# typically be found with
......@@ -94,7 +97,7 @@ SWIGEXEC=/opt/local/bin/swig
PYTHONCFLAGS=-I/System/Library/Frameworks/Python.framework/Versions/2.7/include/python2.7 \
-I/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/numpy/core/include
PYTHONLIB=-lpython
##########################################################################
# Faiss GPU
......
gpu/Makefile
View file @ cf18101f
This diff is collapsed.
gpu/depend 0 → 100644
View file @ cf18101f
This diff is collapsed.
gpu/tests/Makefile 0 → 100644
View file @ cf18101f
# Copyright (c) 2015-present, Facebook, Inc.
# All rights reserved.
#
# This source code is licensed under the BSD+Patents license found in the
# LICENSE file in the root directory of this source tree.
-include ../../makefile.inc
demo_ivfpq_indexing_gpu: demo_ivfpq_indexing_gpu.cpp ../../libfaiss.a ../libgpufaiss.a
$(NVCC) $(NVCCFLAGS) -o $@ $^ $(NVCCLDFLAGS) $(LIBS) $(NVCCLIBS)
clean:
rm -f demo_ivfpq_indexing_gpu
.PHONY: clean
makefile.inc.in 0 → 100644
View file @ cf18101f
# Copyright (c) 2015-present, Facebook, Inc.
# All rights reserved.
#
# This source code is licensed under the BSD+Patents license found in the
# LICENSE file in the root directory of this source tree.
CXX = @CXX@
CXXCPP = @CXXCPP@
# NOTE: Change to `-DFINTEGER=long` if using MKL.
# TODO: Investigate the LAPACKE wrapper for LAPACK, which defines the correct
# type for FORTRAN integers.
CPPFLAGS = -DFINTEGER=int @CPPFLAGS@
CXXFLAGS = -fPIC @OPENMP_CXXFLAGS@ -m64 -Wno-sign-compare @CXXFLAGS@
CPUFLAGS = -msse4 -mpopcnt
LDFLAGS = @OPENMP_CXXFLAGS@ @LDFLAGS@
LIBS = @BLAS_LIBS@ @LAPACK_LIBS@ @LIBS@
PYTHONCFLAGS = @PYTHON_CFLAGS@ -I@NUMPY_INCLUDE@
NVCC = @NVCC@
NVCCLDFLAGS = @NVCC_LDFLAGS@
NVCCLIBS = @NVCC_LIBS@
CUDAROOT = @CUDA_PREFIX@
CUDACFLAGS = @NVCC_CPPFLAGS@
NVCCFLAGS = -I $(CUDAROOT)/targets/x86_64-linux/include/ \
-Xcompiler -fPIC \
-Xcudafe --diag_suppress=unrecognized_attribute \
-gencode arch=compute_35,code="compute_35" \
-gencode arch=compute_52,code="compute_52" \
-gencode arch=compute_60,code="compute_60" \
-lineinfo \
-ccbin $(CXX) -DFAISS_USE_FLOAT16
OS = $(shell uname -s)
SHAREDEXT = so
SHAREDFLAGS = -shared
ifeq ($(OS),Darwin)
SHAREDEXT = dylib
SHAREDFLAGS = -dynamiclib -undefined dynamic_lookup
endif
MKDIR_P = @MKDIR_P@
PYTHON = @PYTHON@
SWIG = @SWIG@
prefix ?= @prefix@
exec_prefix ?= @exec_prefix@
libdir = @libdir@
includedir = @includedir@
python/Makefile 0 → 100644
View file @ cf18101f
# Copyright (c) 2015-present, Facebook, Inc.
# All rights reserved.
#
# This source code is licensed under the BSD+Patents license found in the
# LICENSE file in the root directory of this source tree.
-include ../makefile.inc
HEADERS = $(wildcard ../*.h)
all: cpu
#############################
# CPU
cpu: _swigfaiss.so
# Also silently generates swigfaiss.py.
swigfaiss_wrap.cpp: swigfaiss.swig $(HEADERS)
$(SWIG) -python -c++ -Doverride= -I../ -o $@ $<
swigfaiss_wrap.o: swigfaiss_wrap.cpp
$(CXX) $(CXXFLAGS) $(CPUFLAGS) $(PYTHONCFLAGS) -I../ -c $< -o $@
# Extension is .so even on OSX.
_swigfaiss.so: swigfaiss_wrap.o ../libfaiss.a
$(CXX) $(SHAREDFLAGS) $(LDFLAGS) -o $@ $^ $(LIBS)
#############################
# GPU
gpu: _swigfaiss_gpu.so
# Also silently generates swigfaiss.py.
swigfaiss_gpu_wrap.cpp: swigfaiss.swig
$(SWIG) -python -c++ -Doverride= -I../ -DGPU_WRAPPER -o $@ $<
swigfaiss_gpu_wrap.o: swigfaiss_gpu_wrap.cpp
$(NVCC) $(NVCCFLAGS) $(PYTHONCFLAGS) -I../ -c $< -o $@
_swigfaiss_gpu.so: swigfaiss_gpu_wrap.o ../gpu/libgpufaiss.a ../libfaiss.a
$(CXX) $(SHAREDFLAGS) $(NVCCLDFLAGS) -o $@ $^ $(NVCCLIBS)
clean:
rm -f swigfaiss_wrap.cpp swigfaiss_gpu_wrap.cpp
rm -f swigfaiss.py swigfaiss_gpu.py
rm -f _swigfaiss.so _swigfaiss_gpu.so
test: _swigfaiss.so
PYTHONPATH=./ $(PYTHON) -m unittest discover tests/ -v
.PHONY: all clean cpu gpu test
python/swigfaiss.py
View file @ cf18101f
This diff is collapsed.
python/swigfaiss_gpu.py
View file @ cf18101f
This diff is collapsed.
tests/Makefile
View file @ cf18101f
MAKEFILE_INC=../makefile.inc
# Copyright (c) 2015-present, Facebook, Inc.
# All rights reserved.
#
# This source code is licensed under the BSD+Patents license found in the
# LICENSE file in the root directory of this source tree.
-include $(MAKEFILE_INC)
-include ../makefile.inc
CC_TESTS = test_ivfpq_indexing test_ivfpq_codec test_merge test_ondisk_ivf test_pairs_decoding
TESTS_SRC = $(wildcard *.cpp)
TESTS = $(TESTS_SRC:.cpp=.o)
all: $(CC_TESTS) ../py
all: run
run: all
./run_tests.sh
run: tests
./tests
%: %.cpp ../libfaiss.a gtest/make/gtest_main.a
$(CC) $(CFLAGS) -o $@ $^ $(LDFLAGS) $(BLASLDFLAGS) \
-Igtest/include -I../..
tests: $(TESTS) ../libfaiss.a gtest/make/gtest_main.a
$(CXX) -o $@ $^ $(LDFLAGS) $(LIBS)
../libfaiss.a:
cd ../ && make libfaiss.a
../py:
cd ../ && make py
%.o: %.cpp gtest
$(CXX) $(CPPFLAGS) $(CXXFLAGS) $(CPUFLAGS) -c -o $@ $< -Igtest/include -I../..
gtest/make/gtest_main.a: gtest
cd gtest/make && make CXX=$(CC) gtest_main.a
make -C gtest/make CXX="$(CXX)" CXXFLAGS="$(CXXFLAGS)" gtest_main.a
gtest:
curl -L https://github.com/google/googletest/archive/release-1.8.0.tar.gz | tar xz && \
......@@ -28,4 +29,9 @@ gtest:
rm -rf googletest-release-1.8.0
clean:
rm -rf $(CC_TESTS) gtest
rm -f test_runner
rm -f *.o
rm -rf gtest
.PHONY: all clean run
tests/run_tests.sh deleted 100755 → 0
View file @ dcd83f95
#!/bin/sh
PYTHON=${PYTHON:-"python"}
make
for t in test_*.py; do
PYTHONPATH=.. $PYTHON $t || exit 1
done
for t in test_*.cpp; do
if [ "$t" != "test_blas.cpp" ]; then
./${t%.*} || exit 1
fi
done
tests/test_ivfpq_codec.cpp
View file @ cf18101f
......@@ -16,6 +16,8 @@
#include <faiss/utils.h>
namespace {
// dimension of the vectors to index
int d = 64;
......@@ -43,6 +45,7 @@ double eval_codec_error (long ncentroids, long m, const std::vector<float> &v)
return faiss::fvec_L2sqr (v.data(), v2.data(), nb * d);
}
} // namespace
TEST(IVFPQ, codec) {
......
tests/test_merge.cpp
View file @ cf18101f
......@@ -20,21 +20,21 @@
#include <faiss/OnDiskInvertedLists.h>
namespace faiss {
namespace {
// Main function to test
// Merge index1 into index0. Works on IndexIVF's and IndexIVF's
// embedded in a IndexPreTransform
void merge_into(Index *index0, Index *index1, bool shift_ids) {
void merge_into(faiss::Index *index0, faiss::Index *index1, bool shift_ids) {
FAISS_THROW_IF_NOT (index0->d == index1->d);
IndexIVF *ivf0 = dynamic_cast<IndexIVF *>(index0);
IndexIVF *ivf1 = dynamic_cast<IndexIVF *>(index1);
faiss::IndexIVF *ivf0 = dynamic_cast<faiss::IndexIVF *>(index0);
faiss::IndexIVF *ivf1 = dynamic_cast<faiss::IndexIVF *>(index1);
if (!ivf0) {
IndexPreTransform *pt0 = dynamic_cast<IndexPreTransform *>(index0);
IndexPreTransform *pt1 = dynamic_cast<IndexPreTransform *>(index1);
faiss::IndexPreTransform *pt0 = dynamic_cast<faiss::IndexPreTransform *>(index0);
faiss::IndexPreTransform *pt1 = dynamic_cast<faiss::IndexPreTransform *>(index1);
// minimal sanity check
FAISS_THROW_IF_NOT (pt0 && pt1);
......@@ -43,8 +43,8 @@ void merge_into(Index *index0, Index *index1, bool shift_ids) {
FAISS_THROW_IF_NOT (typeid(pt0->chain[i]) == typeid(pt1->chain[i]));
}
ivf0 = dynamic_cast<IndexIVF *>(pt0->index);
ivf1 = dynamic_cast<IndexIVF *>(pt1->index);
ivf0 = dynamic_cast<faiss::IndexIVF *>(pt0->index);
ivf1 = dynamic_cast<faiss::IndexIVF *>(pt1->index);
}
FAISS_THROW_IF_NOT (ivf0);
......@@ -57,8 +57,6 @@ void merge_into(Index *index0, Index *index1, bool shift_ids) {
index1->ntotal = ivf1->ntotal;
}
};
struct Tempfilename {
......@@ -184,6 +182,8 @@ int compare_merged (faiss::IndexShards *index_shards, bool shift_ids,
return ndiff;
}
} // namespace
// test on IVFFlat with implicit numbering
TEST(MERGE, merge_flat_no_ids) {
......
tests/test_ondisk_ivf.cpp
View file @ cf18101f
......@@ -23,6 +23,7 @@
#include <faiss/index_io.h>
namespace {
struct Tempfilename {
......@@ -50,6 +51,8 @@ struct Tempfilename {
pthread_mutex_t Tempfilename::mutex = PTHREAD_MUTEX_INITIALIZER;
} // namespace
TEST(ONDISK, make_invlists) {
int nlist = 100;
......
tests/test_pairs_decoding.cpp
View file @ cf18101f
......@@ -19,6 +19,10 @@
#include <faiss/VectorTransform.h>
namespace {
typedef faiss::Index::idx_t idx_t;
/*************************************************************
* The functions to test, that can be useful in FANN
*************************************************************/
......@@ -34,7 +38,7 @@
*/
void Search_centroid(faiss::Index *index,
const float* embeddings, int num_objects,
int64_t* centroid_ids)
idx_t* centroid_ids)
{
const float *x = embeddings;
std::unique_ptr<float[]> del;
......@@ -65,9 +69,9 @@ void search_and_retrun_centroids(faiss::Index *index,
const float* xin,
long k,
float *distances,
int64_t* labels,
int64_t* query_centroid_ids,
int64_t* result_centroid_ids)
idx_t* labels,
idx_t* query_centroid_ids,
idx_t* result_centroid_ids)
{
const float *x = xin;
std::unique_ptr<float []> del;
......@@ -80,7 +84,7 @@ void search_and_retrun_centroids(faiss::Index *index,
assert(index_ivf);
size_t nprobe = index_ivf->nprobe;
std::vector<long> cent_nos (n * nprobe);
std::vector<idx_t> cent_nos (n * nprobe);
std::vector<float> cent_dis (n * nprobe);
index_ivf->quantizer->search(
n, x, nprobe, cent_dis.data(), cent_nos.data());
......@@ -95,7 +99,7 @@ void search_and_retrun_centroids(faiss::Index *index,
distances, labels, true);
for (size_t i = 0; i < n * k; i++) {
int64_t label = labels[i];
idx_t label = labels[i];
if (label < 0) {
if (result_centroid_ids)
result_centroid_ids[i] = -1;
......@@ -166,7 +170,7 @@ bool test_Search_centroid(const char *index_key) {
vectors and make sure that each vector does indeed appear in
the inverted list corresponding to its centroid */
std::vector<int64_t> centroid_ids (nb);
std::vector<idx_t> centroid_ids (nb);
Search_centroid(index.get(), xb.data(), nb, centroid_ids.data());
const faiss::IndexIVF * ivf = get_IndexIVF(index.get());
......@@ -192,7 +196,7 @@ int test_search_and_return_centroids(const char *index_key) {
std::vector<float> xb = make_data(nb); // database vectors
auto index = make_index(index_key, xb);
std::vector<int64_t> centroid_ids (nb);
std::vector<idx_t> centroid_ids (nb);
Search_centroid(index.get(), xb.data(), nb, centroid_ids.data());
faiss::IndexIVF * ivf = get_IndexIVF(index.get());
......@@ -204,17 +208,17 @@ int test_search_and_return_centroids(const char *index_key) {
// compute a reference search result
std::vector<long> refI (nq * k);
std::vector<idx_t> refI (nq * k);
std::vector<float> refD (nq * k);
index->search (nq, xq.data(), k, refD.data(), refI.data());
// compute search result
std::vector<long> newI (nq * k);
std::vector<idx_t> newI (nq * k);
std::vector<float> newD (nq * k);
std::vector<int64_t> query_centroid_ids (nq);
std::vector<int64_t> result_centroid_ids (nq * k);
std::vector<idx_t> query_centroid_ids (nq);
std::vector<idx_t> result_centroid_ids (nq * k);
search_and_retrun_centroids(index.get(),
nq, xq.data(), k,
......@@ -253,6 +257,9 @@ int test_search_and_return_centroids(const char *index_key) {
return 0;
}
} // namespace
/*************************************************************
* Test entry points
*************************************************************/
......
tutorial/cpp/Makefile
View file @ cf18101f
MAKEFILE_INC=../../makefile.inc
# Copyright (c) 2015-present, Facebook, Inc.
# All rights reserved.
#
# This source code is licensed under the BSD+Patents license found in the
# LICENSE file in the root directory of this source tree.
-include $(MAKEFILE_INC)
-include ../../makefile.inc
NVCCLDFLAGS = -Xcompiler \"-Wl,-rpath=../../:../../gpu/\" \
-L../.. -L../../gpu -lfaiss -lgpufaiss
CPU_TARGETS = 1-Flat 2-IVFFlat 3-IVFPQ
GPU_TARGETS = 4-GPU 5-Multiple-GPUs
LDFLAGS = -L../.. -Wl,-rpath=../.. -lfaiss
default: cpu
all: cpu gpu
cpu: 1-Flat 2-IVFFlat 3-IVFPQ
cpu: $(CPU_TARGETS)
gpu: 4-GPU 5-Multiple-GPUs
gpu: $(GPU_TARGETS)
1-Flat: 1-Flat.cpp ../../libfaiss.$(SHAREDEXT)
$(CXX) -o $@ $(CXXFLAGS) $< -I../../../ $(LDFLAGS)
$(CPU_TARGETS): %: %.cpp ../../libfaiss.a
$(CXX) $(CXXFLAGS) -o $@ $^ $(LDFLAGS) -I../../.. $(LIBS)
2-IVFFlat: 2-IVFFlat.cpp ../../libfaiss.$(SHAREDEXT)
$(CXX) -o $@ $(CXXFLAGS) $< -I../../../ $(LDFLAGS)
3-IVFPQ: 3-IVFPQ.cpp ../../libfaiss.$(SHAREDEXT)
$(CXX) -o $@ $(CXXFLAGS) $< -I../../../ $(LDFLAGS)
4-GPU: 4-GPU.cpp ../../libfaiss.$(SHAREDEXT) ../../gpu/libgpufaiss.$(SHAREDEXT)
$(NVCC) $(NVCCFLAGS) -o $@ $< $(NVCCLDFLAGS) -I../../../
5-Multiple-GPUs: 5-Multiple-GPUs.cpp ../../libfaiss.$(SHAREDEXT) \
../../gpu/libgpufaiss.$(SHAREDEXT)
$(NVCC) $(NVCCFLAGS) -o $@ $< $(NVCCLDFLAGS) -I../../../
../../libfaiss.$(SHAREDEXT):
cd ../../ && make libfaiss.$(SHAREDEXT)
../../gpu/libgpufaiss.$(SHAREDEXT):
cd ../../gpu/ && make libgpufaiss.$(SHAREDEXT)
$(GPU_TARGETS): %: %.cpp ../../libfaiss.a ../../gpu/libgpufaiss.a
$(NVCC) $(NVCCFLAGS) -o $@ $^ $(NVCCLDFLAGS) -I../../.. $(NVCCLIBS)
clean:
rm -f 1-Flat 2-IVFFlat 3-IVFPQ 4-GPU 5-Multiple-GPUs
rm -f $(CPU_TARGETS) $(GPU_TARGETS)
.PHONY: all cpu default gpu clean
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment