new roadmap with crawler and simplex chatrooms and servers

README.md

@@ -111,12 +111,26 @@ V1.0.4:
 - py: add an optional way to run lantern.py without any manual inputs by passing arguments (ex: python3 lantern.py 1 name desc link "description")  or simply (python3 lantern.py 4) to synchronize links --> for all options! either manual lantern.py or prompt-less lantern.py with arguments
 - docker: figure out how to dockerize the darknet lantern project while maintaining the onion-only requirement (c0mmando is on it, will merge it when he finishes)
 
-V1.1.0+:
+
+V1.1.0 (SimpleX chatrooms and servers uptime ):
+- V1.1.0: find a way to check if the simplex invite link is still joinable or not, but without joining it every time (it should only join it ONCE) -> make it remember that this invite link equates to this chat
+room name ? and then if chatroom name exists then instead of trying to join it make it look for "this chatroom has been deleted" message, if it doesn't exist then assume the chatroom is still joinable. (problem is that the bot should not join every chatroom every 3 hours just to check the uptime, it pollutes chatrooms by doing that.)
+- V1.1.1: using regex alone, create the functions "isSimpleXChatroomValid", and "IsSimplexServerValid", and it should return True for ALL the different syntaxes for simplex chatroom invite links, and smp and xftp servers
+- V1.1.2: uptime.py: make it able to check the uptime of 1) onion links, 2) simplex chatroom links (ONLY if it can query the simplex daemon on 127.0.0.1:3030), and 3) simplex smp and xftp servers
+
+V1.2.0 SimpleX Crawler:
+- V1.2.0: crawler.py: list all the existing simplex chatroom links in verified.csv, make the simplex bot join them via 127.0.0.1:3030 ONLY if it didn't join that chatroom yet. make sure that crawler.py is able
+ to know which chatroom name he's in corresponds to which invite link (either via simplex directly, or via an external simplex-crawl.csv file (columns: link (simplex:/#contact[...]), joined (y/n))
+- V1.2.1: crawler.py: find a way to list (in python) the last unread messages, list them one by one, AND in each message the bot should be able to find, find every simplex link, every .onion link, and every simplex smp / xftp servers links, even if there are multiple onion / simplex links in the same message, he should be able to recognize all of them and list them locally in a crawled.csv file (for now at least, later it'll be in unverified.csv directly)
+- V1.2.2: crawler.py: make the script categorize the onion links into "onion websites", the simplex chatroom invite links into "simplex chatrooms", and the simplex servers smp and xftp links into "simplex serv
+ers" categories, AND in unverified.csv directly
+
+V1.4.0+ PGP support:
 - csv+php+py: implement PGP support to list public pgp keys for verified websites
 - csv+php: figure out how to expand the software to include simplex chatrooms (maybe add another column ?) 
 - py: simplex chatrooms, figure out a way to check their uptime aswell (if the invite link is still valid or not)
 
-V1.2.0+:
+V1.5.0+ I2P support:
 - add i2p support for eepsites (hopefully by then the blogposts on i2p will be written)
 ``` 
 

SimpleX/README.md

@@ -0,0 +1,76 @@
+## HOW TO SETUP ##
+
+This is the code to check if a simplex link is valid and joinable.
+Steps to make the script work:
+1) Download the simplex-chat cli client. You can follow this guide  https://simplex.chat/docs/cli.html#download-chat-client.
+
+wget -qO- https://raw.githubusercontent.com/simplex-chat/simplex-chat/stable/install.sh | bash
+source ~/.zshrc
+fix libcrypto.so.1.1 cannot open shared object file as described here https://github.com/simplex-chat/simplex-chat/issues/1274 :
+sudo apt install libssl-dev
+sudo ln -s /usr/lib/x86_64-linux-gnu/libcrypto.so /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1
+(doesnt fix it) so instead we download the simplex binary:
+
+wget https://github.com/simplex-chat/simplex-chat/releases/latest/download/simplex-chat-ubuntu-22_04-x86-64
+chmod +x ./simplex-chat-ubuntu-22_04-x86-64
+
+
+2) Create an account for the bot -> simplex-chat-ubuntu-22_04-x86-64 -d $HOME/.simplex/simplep-bot -x
+   Use -x flag to use tor.
+
+[ laptop-privateVM ] [ /dev/pts/8 ] [Documents/darknet-lantern.sovereign/simpleP]
+→ ./simplex-chat-ubuntu-22_04-x86-64  -d $HOME/.simplex/simplep-bot -x
+SimpleX Chat v6.3.0.8
+db: /home/nihilist/.simplex/simplep-bot_chat.db, /home/nihilist/.simplex/simplep-bot_agent.db
+using SOCKS5 proxy SocksProxyWithAuth SocksIsolateByAuth 127.0.0.1:9050 for ALL servers.
+private message routing mode: unknown, fallback: protected
+type "/help" or "/h" for usage info
+/h
+No user profiles found, it will be created now.
+Please choose your display name.
+It will be sent to your contacts when you connect.
+It is only stored on your device and you can change it later.
+display name: invalid display name: /h
+you could use this one: h
+
+display name: Lantern
+Current user: Lantern
+                             __   __
+  ___ ___ __  __ ___ _    ___\ \ / / ___ _  _   _ _____
+ / __|_ _|  \/  | _ \ |  | __ \ V / / __| || | /_\_   _|
+ \__ \| || |\/| |  _/ |__| _| / . \| (__| __ |/ _ \| |
+ |___/___|_|  |_|_| |____|___/_/ \_\\___|_||_/_/ \_\_|
+
+Welcome Lantern!
+Thank you for installing SimpleX Chat!
+
+Connect to SimpleX Chat developers for any questions - just type /simplex
+
+Type /help for usage info, /welcome to show this message
+>
+
+#BEGIN SETUP OF THE SIMPLEX CLIENT:
+
+/smp smp://BD4qkVq8lJUgjHt0kUaxeQBYsKaxDejeecxm6-2vOwI=@b6geeakpwskovltbesvy3b6ah3ewxfmnhnshojndmpp7wcv2df7bnead.onion
+ok
+
+/xftp xftp://emX7ForsbdpIscNiDZ6b0HTbfFUayn00C1wmeVTofYA=@wg54vc6p3dscshywvt2wninachqoarrodtunapds7t7p47sn5e3qonid.onion:5233
+ok
+
+/network socks=on
+using SOCKS5 proxy SocksProxyWithAuth SocksIsolateByAuth 127.0.0.1:9050 for ALL servers.
+
+
+3) Create a virtual environment and install websockets module \
+   sudo apt update -y ; sudo apt install python3-venv
+   python3 -m venv my_venv
+   source my_venv/bin/activate
+   pip install websockets
+
+4) Start a websocket(chat server) -> simplex-chat-ubuntu-22_04-x86-64 -x -p 3030 -d simplep-bot
+   You can use any port number. Make sure to change the port number in the script as well. Here we are using port 3030.
+
+[ laptop-privateVM ] [ /dev/pts/8 ] [Documents/darknet-lantern.sovereign/simpleP]
+→ ./simplex-chat-ubuntu-22_04-x86-64  -d $HOME/.simplex/simplep-bot -x -p 3030
+
+5) Call the is_simplex_link_valid(`link`) function.

SimpleX/main.py

@@ -0,0 +1,32 @@
+from websockets.sync.client import connect
+import json
+import random
+
+
+
+def is_simplex_link_valid(simplex_link: str) -> bool:
+    """
+    Connects to the group using the `simplex_link`. If the response contains error False will be returned else True.
+    """
+    with connect("ws://localhost:3030") as websocket:
+        query = f"/c incognito {simplex_link}"
+        command = {
+            'corrId': f"id{random.randint(0,999999)}",
+            'cmd': query,
+        }
+        websocket.send(json.dumps(command))
+        message = websocket.recv()
+        response = json.loads(message)
+        response_type = response['resp']['type']
+        # print(response)
+        if 'error' in response_type.lower():
+            return False
+        elif 'deleted' in response_type.lower():
+            return False
+
+        print('[+] Success!',response_type)
+        return True
+
+if __name__ == '__main__':
+        sxclink='https://simplex.chat/contact#/?v=2-7&smp=smp%3A%2F%2FBD4qkVq8lJUgjHt0kUaxeQBYsKaxDejeecxm6-2vOwI%3D%40b6geeakpwskovltbesvy3b6ah3ewxfmnhnshojndmpp7wcv2df7bnead.onion%2F4woLIDlpkvXRvZmaAiWA802OwiyxekdJ%23%2F%3Fv%3D1-3%26dh%3DMCowBQYDK2VuAyEAzIAoE-OWDqFJXMqgunIWHPpE_u7e52Wtu8TioPc1QwI%253D&data=%7B%22groupLinkId%22%3A%22Srr1_MNob7WfPTQIY-ug5Q%3D%3D%22%7D'
+        is_simplex_link_valid(sxclink)

scripts/lantern.py

@@ -259,6 +259,7 @@ Maintenance:
                                 # check if there are no results, dont proceed if there are none!
                                 if filter_uvdf.size == 0:
                                     print_colors("ERROR no results, skipping.",is_error=True)
+                                    break
                                 else:
                                     # Each of the rows has an index,
                                     index=-1