Connection refused when running Grakn in a Docker container


#1

Hey guys, I have a curious issue:

When running Grakn using a Docker container under the same machine as the host machine, Grakn refuses the connection between the Java client running on the host machine and Grakn (running on the Docker container).

The Grakn dashboard is perfectly working and the pertinent ports are correctly mapped so I can access it on the host machine.

Is there something I’m missing or something I can do?

Kind regards,

Daniel.


#2

Hi @limiaspasdaniel!

Thanks for trying out the dockerised Grakn!

If you’re writing your own Java app using Grakn Java API then it’s most likely trying to connect to the Cassandra ports which are not exposed at all. I’ve just pushed a new version with exposed port 9042/9160. Can you pull grakn:latest and see if it works?

If it still doesn’t work, can you paste the full error message you’re getting?

Cheers,
Thanh


#3

I just realised that cassandra within the docker container only listens to localhost so it won’t work unless you edit /grakn/conf/cassandra/cassandra.yaml within the container.
The two options to edit are listen_address and rpc_address.

You will also need to map the new ports by running:

docker run -dt -p 4567:4567 -p 9042:9042 -p 9160:9160 --name grakn-0.11.0 graknlabs/grakn:0.11.0

#4

Hi @thanh!

Thank you for the quick response.

I run the docker command you mentioned but as I’m pretty new using it I still don’t know which values should I set for the listen_address and rpc_address variables in the cassandra.yaml.

Could you please give me a hand?

Thank you very much,

Daniel.


#6

Did you manage to get this working?
You need to set them to the IP of the docker container.


#7

Hi @thanh.

I set them to the IP of the docker container, and I got some improvements but still no luck. Now the previous error message :“Connection refused” is not showing anymore. However this one shows:

21:28:05.170 [main] DEBUG com.thinkaurelius.titan.diskstorage.cassandra.CassandraTransaction - Created CassandraTransaction@379ab47b[read=QUORUM,write=QUORUM]
21:28:15.240 [pool-4-thread-1] DEBUG com.netflix.astyanax.thrift.ThriftConverter - java.net.SocketTimeoutException: connect timed out
21:28:15.334 [main] DEBUG com.netflix.astyanax.connectionpool.impl.AbstractHostPartitionConnectionPool - Retry policy[com.netflix.astyanax.retry.BoundedExponentialBackoff@6f70f32f[maxSleepTimeMs=25000,MAX_SHIFT=30,random=java.util.Random@548e76f1,baseSleepTimeMs=100,maxAttempts=8,attempts=1]] will allow a subsequent retry for operation [class com.netflix.astyanax.thrift.ThriftColumnFamilyQueryImpl$4$1] on keyspace [graknsystem] on pinned host[null]
21:28:25.535 [pool-4-thread-1] DEBUG com.netflix.astyanax.thrift.ThriftConverter - java.net.SocketTimeoutException: connect timed out
21:28:25.835 [main] DEBUG com.netflix.astyanax.connectionpool.impl.AbstractHostPartitionConnectionPool - Retry policy[com.netflix.astyanax.retry.BoundedExponentialBackoff@6f70f32f[maxSleepTimeMs=25000,MAX_SHIFT=30,random=java.util.Random@548e76f1,baseSleepTimeMs=100,maxAttempts=8,attempts=3]] will allow a subsequent retry for operation [class com.netflix.astyanax.thrift.ThriftColumnFamilyQueryImpl$4$1] on keyspace [graknsystem] on pinned host[null]
21:28:36.837 [pool-4-thread-1] DEBUG com.netflix.astyanax.thrift.ThriftConverter - java.net.SocketTimeoutException: connect timed out

Thank you,

Daniel


#8

Hey @limiaspasdaniel

I think one more setting needs to be changed in /grakn/conf/main/grakn.properties:

storage.hostname=127.0.0.1

to the same IP set in cassandra.yaml earlier.

I’ve just changed this and tested with Graql shell, which is a Java app, from outside the container and it seems to work.

If you’re still having trouble with the container, I recommend trying our normal distribution package file from Github (https://github.com/graknlabs/grakn/releases). This should work out of the box if you’re writing a Java app as opposed to only using the Visualiser.


#9

Hi @thanh,

I cannot use the grakn binaries as I’m using Windows.
When also changing the value of storage.hostname with happens to be 172.17.0.2 (I got the value by getting the IPAddress value of the output generated by running docker inspect ) it just says:
java.net.SocketTimeoutException: connect timed out

So to sum up, I changed the listen_address and rpc_address in cassandra.yaml, and the storage.hostname in grakn.properties. Restarted the docker container and checked out using the “cat” command if the files were properly updated. Afterwards this is the complete stacktrace when executing the client in the host machine by calling in a Java app the next method:
Grakn.factory(Grakn.DEFAULT_URI, "grakn").getGraph();

The console output:

Connected to the target VM, address: '127.0.0.1:53720', transport: 'socket'
Please start Grakn Engine
You can get more information on how to do so using our setup guide: https://grakn.ai/pages/documentation/get-started/setup-guide.html
15:15:59.538 [main] DEBUG com.thinkaurelius.titan.diskstorage.cassandra.astyanax.AstyanaxStoreManager - About to instantiate class public com.netflix.astyanax.connectionpool.impl.FixedRetryBackoffStrategy(int,int) with 2 arguments
15:15:59.563 [main] DEBUG com.thinkaurelius.titan.diskstorage.cassandra.astyanax.AstyanaxStoreManager - Instantiated RetryBackoffStrategy object com.netflix.astyanax.connectionpool.impl.FixedRetryBackoffStrategy@1613674b from config string "com.netflix.astyanax.connectionpool.impl.FixedRetryBackoffStrategy,1000,5000"
15:15:59.570 [main] DEBUG com.thinkaurelius.titan.diskstorage.cassandra.astyanax.AstyanaxStoreManager - About to instantiate class public com.netflix.astyanax.retry.BoundedExponentialBackoff(long,long,int) with 3 arguments
15:15:59.571 [main] DEBUG com.thinkaurelius.titan.diskstorage.cassandra.astyanax.AstyanaxStoreManager - Instantiated RetryPolicy object com.netflix.astyanax.retry.BoundedExponentialBackoff@33990a0c[maxSleepTimeMs=25000,MAX_SHIFT=30,random=java.util.Random@3059cbc,baseSleepTimeMs=100,maxAttempts=8,attempts=0] from config string "com.netflix.astyanax.retry.BoundedExponentialBackoff,100,25000,8"
15:15:59.624 [main] DEBUG com.thinkaurelius.titan.diskstorage.cassandra.astyanax.AstyanaxStoreManager - Custom RetryBackoffStrategy com.netflix.astyanax.connectionpool.impl.FixedRetryBackoffStrategy@1613674b
15:16:00.848 [main] INFO com.netflix.astyanax.connectionpool.impl.ConnectionPoolMBeanManager - Registering mbean: com.netflix.MonitoredResources:type=ASTYANAX,name=ClusterTitanConnectionPool,ServiceType=connectionpool
15:16:00.906 [main] INFO com.netflix.astyanax.connectionpool.impl.CountingConnectionPoolMonitor - AddHost: 172.17.0.2
15:16:10.948 [main] DEBUG com.thinkaurelius.titan.diskstorage.cassandra.astyanax.AstyanaxStoreManager - Failed to describe keyspace graknsystem
15:16:10.948 [main] DEBUG com.thinkaurelius.titan.diskstorage.cassandra.astyanax.AstyanaxStoreManager - Creating keyspace graknsystem...
15:16:10.956 [pool-4-thread-1] DEBUG com.netflix.astyanax.thrift.ThriftConverter - java.net.SocketTimeoutException: connect timed out
15:16:21.000 [pool-4-thread-1] DEBUG com.netflix.astyanax.thrift.ThriftConverter - java.net.SocketTimeoutException: connect timed out
15:16:21.000 [main] DEBUG com.thinkaurelius.titan.diskstorage.cassandra.astyanax.AstyanaxStoreManager - Failed to create keyspace graknsystem
Exception in thread "main" java.lang.IllegalArgumentException: Graph Factory [ai.grakn.factory.TitanInternalFactory] is not valid
	at ai.grakn.factory.FactoryBuilder.newFactory(FactoryBuilder.java:101)
	at ai.grakn.factory.FactoryBuilder.getGraknGraphFactory(FactoryBuilder.java:82)
	at ai.grakn.factory.FactoryBuilder.getFactory(FactoryBuilder.java:62)
	at ai.grakn.factory.GraknGraphFactoryImpl.configureGraphFactoryRemote(GraknGraphFactoryImpl.java:126)
	at ai.grakn.factory.GraknGraphFactoryImpl.configureGraphFactory(GraknGraphFactoryImpl.java:103)
	at ai.grakn.factory.GraknGraphFactoryImpl.getConfiguredFactory(GraknGraphFactoryImpl.java:81)
	at ai.grakn.factory.GraknGraphFactoryImpl.getGraph(GraknGraphFactoryImpl.java:69)
	at limia.Connection.DBConnection.load(DBConnection.java:37)
	at limia.WebServer.Main.main(Main.kt:17)
Caused by: java.lang.reflect.InvocationTargetException
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
	at ai.grakn.factory.FactoryBuilder.newFactory(FactoryBuilder.java:99)
	... 8 more
Caused by: java.lang.IllegalArgumentException: Could not instantiate implementation: com.thinkaurelius.titan.diskstorage.cassandra.astyanax.AstyanaxStoreManager
	at com.thinkaurelius.titan.util.system.ConfigurationUtil.instantiate(ConfigurationUtil.java:55)
	at com.thinkaurelius.titan.diskstorage.Backend.getImplementationClass(Backend.java:473)
	at com.thinkaurelius.titan.diskstorage.Backend.getStorageManager(Backend.java:407)
	at com.thinkaurelius.titan.graphdb.configuration.GraphDatabaseConfiguration.<init>(GraphDatabaseConfiguration.java:1325)
	at com.thinkaurelius.titan.core.TitanFactory.open(TitanFactory.java:94)
	at com.thinkaurelius.titan.core.TitanFactory.open(TitanFactory.java:84)
	at com.thinkaurelius.titan.core.TitanFactory$Builder.open(TitanFactory.java:139)
	at ai.grakn.factory.TitanInternalFactory.configureGraph(TitanInternalFactory.java:113)
	at ai.grakn.factory.TitanInternalFactory.newTitanGraph(TitanInternalFactory.java:87)
	at ai.grakn.factory.TitanInternalFactory.buildTinkerPopGraph(TitanInternalFactory.java:83)
	at ai.grakn.factory.TitanInternalFactory.buildTinkerPopGraph(TitanInternalFactory.java:56)
	at ai.grakn.factory.AbstractInternalFactory.getTinkerPopGraph(AbstractInternalFactory.java:158)
	at ai.grakn.factory.AbstractInternalFactory.getTinkerPopGraph(AbstractInternalFactory.java:152)
	at ai.grakn.factory.AbstractInternalFactory.getGraph(AbstractInternalFactory.java:123)
	at ai.grakn.factory.AbstractInternalFactory.getGraph(AbstractInternalFactory.java:96)
	at ai.grakn.factory.AbstractInternalFactory.getGraph(AbstractInternalFactory.java:48)
	at ai.grakn.factory.SystemKeyspace.loadSystemOntology(SystemKeyspace.java:117)
	at ai.grakn.factory.FactoryBuilder.newFactory(FactoryBuilder.java:107)
	at ai.grakn.factory.FactoryBuilder.getGraknGraphFactory(FactoryBuilder.java:82)
	at ai.grakn.factory.AbstractInternalFactory.getSystemFactory(AbstractInternalFactory.java:80)
	at ai.grakn.factory.AbstractInternalFactory.<init>(AbstractInternalFactory.java:74)
	at ai.grakn.factory.TitanInternalFactory.<init>(TitanInternalFactory.java:60)
	... 13 more
Caused by: java.lang.reflect.InvocationTargetException
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
	at com.thinkaurelius.titan.util.system.ConfigurationUtil.instantiate(ConfigurationUtil.java:44)
	... 34 more
Caused by: com.thinkaurelius.titan.diskstorage.TemporaryBackendException: Temporary failure in storage backend
	at com.thinkaurelius.titan.diskstorage.cassandra.astyanax.AstyanaxStoreManager.ensureKeyspaceExists(AstyanaxStoreManager.java:580)
	at com.thinkaurelius.titan.diskstorage.cassandra.astyanax.AstyanaxStoreManager.<init>(AstyanaxStoreManager.java:291)
	... 39 more
Caused by: com.netflix.astyanax.connectionpool.exceptions.PoolTimeoutException: PoolTimeoutException: [host=172.17.0.2(172.17.0.2):9160, latency=10001(10001), attempts=1]Timed out waiting for connection
	at com.netflix.astyanax.connectionpool.impl.SimpleHostConnectionPool.waitForConnection(SimpleHostConnectionPool.java:231)
	at com.netflix.astyanax.connectionpool.impl.SimpleHostConnectionPool.borrowConnection(SimpleHostConnectionPool.java:198)
	at com.netflix.astyanax.connectionpool.impl.RoundRobinExecuteWithFailover.borrowConnection(RoundRobinExecuteWithFailover.java:84)
	at com.netflix.astyanax.connectionpool.impl.AbstractExecuteWithFailoverImpl.tryOperation(AbstractExecuteWithFailoverImpl.java:117)
	at com.netflix.astyanax.connectionpool.impl.AbstractHostPartitionConnectionPool.executeWithFailover(AbstractHostPartitionConnectionPool.java:352)
	at com.netflix.astyanax.thrift.ThriftClusterImpl.executeSchemaChangeOperation(ThriftClusterImpl.java:146)
	at com.netflix.astyanax.thrift.ThriftClusterImpl.internalCreateKeyspace(ThriftClusterImpl.java:321)
	at com.netflix.astyanax.thrift.ThriftClusterImpl.addKeyspace(ThriftClusterImpl.java:294)
	at com.thinkaurelius.titan.diskstorage.cassandra.astyanax.AstyanaxStoreManager.ensureKeyspaceExists(AstyanaxStoreManager.java:575)
	... 40 more
Disconnected from the target VM, address: '127.0.0.1:53720', transport: 'socket'

The docker container run command is the default one (The one you updated in the latest Grakn docker container update).

When the Docker container is running, I try to ping the container IP and I get no response. It should be responding to the ping request right?

Thank you for your time,

Daniel


#10

I think this might be a docker networking issue.
Are you using “Docker for Windows” (Hyper-V) or “Docker Toolbox” (VirtualBox)?
If the former then this might help:


or

Also please check your windows firewall is allowing ports 4567/9042/9160 to the docker container network.


#11

All right now! Everything working by following your previous instructions and running
route add 172.17.0.0 mask 255.255.0.0 10.0.75.2 -p on an elevated command prompt.

Thank you very much for your time,

Daniel


#12

Superb! I’m glad it’s all working now.

I will try and get the config files to auto configure so that the only thing required would be to add the route for Docker.


#13

Trying this on Ubuntu using the Docker image (https://hub.docker.com/r/graknlabs/grakn/) and getting the same connection refused issue. I’ve checked that the Cassandra and Grakn configs are set correctly to the container IP. Is there anything else to try? the dashboard and sample queries work fine, it’s only when I run a ‘compute’ query that I see this error e.g. ‘compute count in person;’

			java.io.IOException: Unable to connect to server localhost:9160
					at org.apache.cassandra.hadoop.ConfigHelper.createConnection(ConfigHelper.java:574)
					at org.apache.cassandra.hadoop.ConfigHelper.getClientFromAddressList(ConfigHelper.java:545)
					at org.apache.cassandra.hadoop.ConfigHelper.getClientFromInputAddressList(ConfigHelper.java:529)
					at org.apache.cassandra.hadoop.AbstractColumnFamilyInputFormat.getRangeMap(AbstractColumnFamilyInputFor$
					at org.apache.cassandra.hadoop.AbstractColumnFamilyInputFormat.getSplits(AbstractColumnFamilyInputForma$
					at org.janusgraph.hadoop.formats.cassandra.CassandraBinaryInputFormat.getSplits(CassandraBinaryInputFor$
					at org.janusgraph.hadoop.formats.util.GiraphInputFormat.getSplits(GiraphInputFormat.java:62)
					at org.apache.spark.rdd.NewHadoopRDD.getPartitions(NewHadoopRDD.scala:120)
					at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:239)
					at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:237)
					at scala.Option.getOrElse(Option.scala:120)
					at org.apache.spark.rdd.RDD.partitions(RDD.scala:237)
					at org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:35)
					at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:239)
					at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:237)
					at scala.Option.getOrElse(Option.scala:120)
					at org.apache.spark.rdd.RDD.partitions(RDD.scala:237)
					at org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:35)
					at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:239)
					at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:237)
					at scala.Option.getOrElse(Option.scala:120)
					at org.apache.spark.rdd.RDD.partitions(RDD.scala:237)
					at org.apache.spark.api.java.JavaRDDLike$class.partitions(JavaRDDLike.scala:64)
					at org.apache.spark.api.java.AbstractJavaRDDLike.partitions(JavaRDDLike.scala:46)
					at ai.grakn.kb.internal.computer.GraknSparkComputer.lambda$submitWithExecutor$0(GraknSparkComputer.java$
					at java.util.concurrent.FutureTask.run(FutureTask.java:266)
					at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
					at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
					at java.lang.Thread.run(Thread.java:748)
			Caused by: org.apache.thrift.transport.TTransportException: java.net.ConnectException: Connection refused (Conn$
					at org.apache.thrift.transport.TSocket.open(TSocket.java:187)
					at org.apache.thrift.transport.TFramedTransport.open(TFramedTransport.java:81)
					at org.apache.cassandra.thrift.TFramedTransportFactory.openTransport(TFramedTransportFactory.java:41)
					at org.apache.cassandra.hadoop.ConfigHelper.createConnection(ConfigHelper.java:569)

Not able to run command ./grakn server start on cmd Windows
#14

Hi Tom,

I made some changes for 0.18 that would allow the Java API from outside of Docker to connect directly to the storage layer for Grakn. Unfortunately it looks like that broke analytics.

If you don’t need to connect using the Java API, you can start the container this way for Analytics to work:

docker run -dt -v $(pwd)/db/:/grakn/db/ -p 4567:4567  -p 9042:9042 -p 9160:9160 -e STORAGE_LISTEN_ADDRESS=localhost -e STORAGE_RPC_ADDRESS=localhost graknlabs/grakn

In the meantime, we’ll look at fixing this correctly.
Let me know if it works out for you.

Thanh


#15

Thank you that works fine!

Tom