tue/2IOC0-DBL/TwitterDataAnalytics - 2IOC0 DBL Twitter Data Analytics

Age	Commit message (Collapse)	Author	Files	Lines
2014-05-26	Small bugfixes to the SearchImplHEAD master	Maurice Laveaux	1	-19/+28
	* Convers ids to longs instead of ints * input and output fiel correctly named * RateLimit is handled better. Signed-off-by: Maurice Laveaux <maurice@localhost.localdomain>
2014-05-26	Improved the requester	Maurice Laveaux	2	-13/+28
	* Reads a gzip stream. * Uses the http request date for interval measurement.
2014-05-26	Fixed small bug.	Maurice Laveaux	1	-1/+1

2014-05-22	Correctness fixes	Peter Wu	3	-41/+46
	* Throw interrupted exception when interrupted instead of swallowing it, otherwise shutdown will take a longer time. * Decrease number of tweets that must be requested after confirming that the request was successful. * Style: convert to foreach; white-space. * Properly terminate if stop is requested. * Make executeRequest take care of re-queing requests if needed.
2014-05-22	Properly close files after load/save	Peter Wu	1	-16/+14

2014-05-22	Rename getResp to getJson	Peter Wu	3	-14/+14

2014-05-22	Removed unused port.	Maurice Laveaux	1	-2/+0
	* I dun goofed.
2014-05-22	Added saving and loaded the search cache.	Maurice Laveaux	1	-12/+56
	* TODO: threads don't exit safe, so cache is not valid.
2014-05-22	Implemented basic search functionality.	Maurice Laveaux	1	-0/+321
	* Usage, search <screen_name> <amount_tweets>.
2014-05-22	Removed all JSON references.	Maurice Laveaux	2	-5/+0

2014-05-22	Refactored JSON to GSON's implementation.	Maurice Laveaux	2	-36/+46
	* Changed JSONException to JsonParseException.
2014-05-22	Updated the Requesters to throw RateLimitException.	Maurice Laveaux	5	-17/+54
	* The exception contains the reset timestamp in unix time.
2014-05-22	Changed Request response to a new Response class.	Maurice Laveaux	5	-62/+83
	* Response contains the JsonElement, tickrate, reset and leftover data. * Removed getJsonRelaxed, because it was not used elsewhere.
2014-05-08	CompressableDataWriter: fix corruption for cfile	Peter Wu	1	-0/+4

2014-05-07	Fix null ptr when close/commit	Peter Wu	1	-1/+8
	commit closes everything via safeClose, so ensure that tweetGenerated re-opens it if necessary. Fixes: ƒ commit € Exception in thread "Thread-7" java.lang.NullPointerException at io.DataWriter.writeObject(DataWriter.java:137) at io.DataWriter.tweetGenerated(DataWriter.java:85) at provider.CompositeResultListener.tweetGenerated(CompositeResultListener.java:51) at io.StreamImpl$Poller.processObject(StreamImpl.java:255) at io.StreamImpl$Poller.run(StreamImpl.java:245) at java.lang.Thread.run(Thread.java:744)
2014-05-07	Display user for tweet, strip whitespace around keyword	Peter Wu	1	-0/+1

2014-05-07	Add flush command	Peter Wu	1	-1/+13

2014-05-07	Get rid of separate profiles	Peter Wu	3	-42/+7
	These are always available from the tweets themselves...
2014-05-07	Allow commit to disconnect for empty keywords, detect dead connection	Peter Wu	1	-6/+6

2014-05-07	Set read timeout for streaming	Peter Wu	1	-0/+3

2014-05-07	Fix deadlock in Poller if worker thread stops/dies	Peter Wu	1	-0/+10

2014-05-07	StreamImpl: make (un)watchKeywords support a list	Peter Wu	1	-4/+20

2014-05-07	Use IOUtils.closeQuietly for less verbosity	Peter Wu	1	-13/+3

2014-05-03	DataWriter: first read, then open for writing	Peter Wu	1	-2/+3
	Prevents occasional "Unexpected end of ZLIB input stream" errors.
2014-05-03	DataWriter: better detect errors in reading data	Peter Wu	1	-6/+13
	Now print the faulty line and line number. Using BufferedReader instead of Scanner also gave me a hint of the error, so let's stick to it.
2014-05-03	CompressableDataWriter: use separate .gz-suffixed file	Peter Wu	1	-91/+72
	Do not use the same name for compressed and uncompressed files, it is confusing and non-standard. Now it'll use a ".gz"-suffixed file, writing all changes from the uncompressed file if requested.
2014-05-03	Display stats when converting an uncompressed file	Peter Wu	1	-0/+18

2014-05-03	More compression fixes	Peter Wu	1	-9/+12
	* Fix magic check (bytes are read as signed numbers...). * Assume that a file can be compressed if the file is too small. * Try to convert a file only if the gzip detection is correctly closed.
2014-05-02	Properly close files and stream on exit	Peter Wu	1	-1/+2

2014-05-02	Fix compression move which actually lost some data	Peter Wu	1	-28/+21
	rename tricks somehow failed, but apparently there is a reliable Files.move() operation. Add some more debugging messages.
2014-05-02	DataWriter should not open files in constructor	Peter Wu	2	-21/+46
	Right now Store instances are created in the constructor of DataWriter. This causes issues, namely that Store instances are unable to use DataWriter in its constructor. The real-world implication is that convertUncompressed in CompressableDataWriter is unusable for the CompressedStore.
2014-05-02	Reject commits without keywords	Peter Wu	1	-0/+4

2014-05-02	Fix typo in DataWriter	Peter Wu	1	-1/+1

2014-05-02	Abort if file cannot be read, don't leak on error	Peter Wu	1	-11/+17

2014-05-02	CompressableDataWriter: Implement conversion from uncompressed	Peter Wu	1	-1/+64

2014-05-02	Support compressed files	Peter Wu	1	-0/+88

2014-05-02	DataWriter: don't leak input fd	Peter Wu	1	-1/+10

2014-05-02	Refactor DataWriter once more to group a file	Peter Wu	1	-37/+79
	The input and output file are closely related, do not burden others with tracking which output and input need to be matched.
2014-05-02	Make DataWriter more generic	Peter Wu	1	-11/+40
	Prepare for compressed file formats, this requires that the data writer can handle compressed files. In theory an implementation can even write stuff over the network instead of a file.
2014-05-02	Don't ignore IOException stupid!	Peter Wu	1	-34/+23
	Files may not be accessible if they are read-only, or if the directory is read-only, or ... whatever. So, let the caller handle IOException in DataWriter. Return a set if it can be read. Do some misc cleanups (remove unused imports, fix docs, remove obsolete TODOs).
2014-05-02	Avoid hang during close	Peter Wu	1	-4/+29
	Apparently the socket operations are blocking in a way such that even interrupt() cannot stop it. Close the socket (which will generate an IOException masked as JSONException) to immediately close the socket. Also do not leak a socket if an error occurred in the Worker constructor (when a Stream connection is set up).
2014-05-01	Add command to print active keywords	Peter Wu	1	-0/+15

2014-05-01	Don't leak connections	Peter Wu	2	-34/+42

2014-05-01	New thread for notifying of new objects	Peter Wu	1	-13/+95
	* Use volatile boolean instead of AtomicBoolean. There will only be one one writer. * Store received objects in a queue instead of printing them. * Rename thread to clarify its I/O purpose. * Create new Thread for notifying observers of new tweets and users.
2014-05-01	Get rid of unused and broken poll function	Peter Wu	1	-9/+0

2014-05-01	Merge branch 'master' of git.lekensteyn.nl:tue/2IOC0-DBL/TwitterDataAnalytics	Maurice Laveaux	1	-5/+11

2014-05-01	Added the target command with options: file, shell.	Maurice Laveaux	2	-4/+19
	* Implemented target file, which uses DataWriter as RequestListener which writes the tweets and profiles. * Added profiles-filename, tweets-filename to the config.
2014-05-01	Close stream when stopping worker, English tweets only please...	Peter Wu	1	-5/+11

2014-05-01	Allow result and exception handler to be changed	Peter Wu	1	-7/+29

2014-05-01	Fix signing for OAuth POST messages, fix search param	Peter Wu	3	-6/+42