# Overview of Sunbird UCI **Context** The world runs on communication, notifications, and nudges. Notifications could be informative text, a reminder, or even a nudge. The use cases may range from a notification to a student on today's homework, to a notification of a new job posting to a job seeker, it can also be a notification to a farmer on the weather forecast or even a notification to an IAS officer about the weekly progress summary of the project. Also, the users are distributed across different communication channels like WhatsApp, SMS, Telegram, Email, In-App, etc. The goal of all the notifications is to capture the attention of the end-user and drive forward a piece of relevant useful information. Doing so may at times mean sending the same or similar conversation over more than one channel, e.g. Farmers receive the weather advisory over SMS, IVRS call, and even local radio. **Problem Statement** As a government or any service provider organization that wants to send the communication to the end users. The organization can send a one-way communication to users like an advisory, reminder, or nudge. Or can create a two-way communication channel like WhatsApp Bot or a chatbot on govt. portal. Each of these modules turns out to be a complex engineering effort, with a high touch point of engineers to make any modifications in the flow or add a new message or channel of communication. **Vision** UCI is created with a vision to make the conversation flow creation configurable to a level, that it can be done by program owners, without the intervention of any engineers at all. At the same time, it also envisions reducing the redundant coding required for enabling the same business logic for different channels. **How UCI Functions** In short, UCI's architecture has created an abstraction layer between the Logic and the communication channels (WhatsApp, SMS, etc) and various providers (Like Gupshup, Twilio, etc.). UCI uses an adapter-based abstraction layer. An adapter has to be created for every channel+provider that is to be used with UCI. Once contributed, it is available for everyone in the community to leverage. **Benefits of UCI** Following are the key benefits of UCI: (not exhaustive) * Provides an abstraction layer to safeguard PII data of users (Does not store or pass on) * Makes new conversation logic creation configurable (possible to be done using a frontend) * Considerable reduction of custom coding required for different channels # UCI use cases Unified Communications Interface (UCI) aims to democratize the use of different communication channels such as WhatsApp, Telegram, SMS, email and more across all governance use cases. ![](https://1242145269-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FnC2042spZ2TIbqdRA37S%2Fuploads%2F42p5I0Jz57aW4Ps9zKGq%2FUCI%20across%20sectors.png?alt=media\&token=7ad69c68-6609-4e25-af23-d671f6deb8e5) Let's try and understand indicative use cases across two sectors - Education and Agriculture. ### UCI for School Education UCI for School Education has the potential to reduce information asymmetry between teachers, students, parents & the government machinery while strengthening the digital learning ecosystem. ![Sample education use case - Student quizzes through UCI chatbot](https://1242145269-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FnC2042spZ2TIbqdRA37S%2Fuploads%2FU3S76q2n6Q6LkvZxmHuz%2FUCI%20for%20education%20-%20quiz.png?alt=media\&token=cef6cc14-fbb4-4503-9916-4e7b1f03542c) The table below covers some of the major and representative use cases that UCI can potentially enable in education:
CategoryAs aI want toSo that
Administrative SurveysGovernment Officerreport progress for educational government initiatives I can inform senior government officers about progress in my jurisdiction
AssessmentsStudentattempt quizzes on UCI BotI can easily practise and prepare for my final examinations
Reports/Dashboards Teacherview student quiz reports on UCI BotI can monitor my students' performance and support them
Content discoveryTeacherview learning content on UCI BotI can improve my keep upskilling myself and support my students in a better way
NudgesStudentreceive remedial learning content based on quiz score on UCI BotI can improve my learning and understanding about key topics
### UCI for Agriculture UCI for Agriculture has the potential of enabling higher incomes for the farmer through access to the right information at the right time. ![Sample agriculture use case - Farmer advisories through UCI chatbot](https://1242145269-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FnC2042spZ2TIbqdRA37S%2Fuploads%2FczHiXrVUSRjMCnCdfuxs%2FUCI%20for%20agriculture%20-%20agroadvisory.png?alt=media\&token=d530bba7-0a8a-4f1b-90cc-78cabb2a8db9) The table below covers some of the major and representative use cases that UCI can potentially enable in agriculture:
CategoryAs aI want toSo that
AlertsFarmeraccess dynamic weather-based crop advisoriesI can plan for any adversaries in advance
Information DisseminationFarmerreceive best farming / breeding practicesI can improve my productivity and thereafter produce and income
SurveysAdministratorunderstand on-field input prices (seeds, fertilizers, etc)I can take corrective action where required
NudgesAdministratorshare actionable steps with poor performing district officersthey can take those steps and improve performance
Similarly there can be multiple use cases across other sectors - skilling, employment, local governance and healthcare among others. # UCI for Education(Case study) **Usage Scenario** Class teacher of Geeta, a class 5 student from Govt. School, Gurgaon, Haryana has shared a number with her. She has asked her to send a WhatsApp message ‘Hi UCIBot’ on this number. She has been told to enter her details on the bot and take a quiz and oral test. Teacher wants her to submit the printout of her scorecard in the next class. **Steps for Geeta:** 1. Geeta sends starting message to the bot 2. Geeta confirms her identity 3. Geeta selects the subject for quiz 4. Geeta answers all the questions in her quiz 5. Bot prompts for oral assessment 6. Geeta follows the instruction and starts to read 7. At the end of the quiz, Geeta gets a link to download her report 8. Geeta takes a printout of the quiz report and submits it to her class teacher (can also share the downloaded report via WhatsApp with her class teacher) **Here is how it works:** ![](https://lh3.googleusercontent.com/-yK1nq1rAXT1ayOqOScKuGd7j7Rq4aUN2LDtGO7qqpB9hxUbGaNyHzt8inmkkCuhAZk300UeHOT1vMgPyQIALETwtHR76qGxo8LKbNsBYqchDoOgkrjECvZdM16VM1j-6qCccsji) 1. Student Initiates the conversation with bot 2. User is mapped, if the user does not exist their profile is created 3. Role based access level for the user is checked 4. Check menu list mapped to user role is checked 5. Menu list is updated (based on user) 6. User is shown the dynamic menu 7. User selects Hindi Quiz 8. Orchestrator initiates the Quiz Transformer 9. Quiz questions are updated 10. Objective quiz is initiated 11. Student submits Quiz answers for objective quiz 12. Oral reading quiz is initiated 13. Student’s audio is recorded and submitted 14. Local language audio to is converted to text using external service 15. Student results for audio test is updated 16. Student text inputs are checked 17. Student’s final result is updated 18. Student’s report card (pdf format) is requested to be generated 19. Orchestrator receives the report card from the PDF generator 20. Additional content links to be shared with the student is requested 21. Content links are updated for the message 22. Student receives the report card with learning content links ***Note:*** *This section provides an example case study. The purpose is to illustrate a set of real-life case studies and not enumerate all possible usage scenarios. It is expected that the user ecosystem will innovate and find more interesting usage scenarios for this simple and unified communication interface.* # Tech Overview This section will help build your basic tech understanding on the workings of Sunbird UCI. The section is structured in 4 parts: [**Glossary**](/learn/tech-overview/glossary) - Introduces the lingo of Sunbird UCI [**High-level Architecture Design**](/learn/tech-overview/high-level-tech-architecture) - Introduction to the High-level Architecture diagram and introduction to the flow of information within [**Design principles**](/learn/tech-overview/design-principles) - This section explains the 5 design principles that are kept at the center of the Sunbird UCI design # Glossary * **Communication channel**: Any channel/medium where the user comes and interacts with the systsem. Example. SMS, Whatsapp, Email, Slack, custom ChatBot etc.
* **Service Provider**: Any authorized business that has the permission to sub-license the APIs of the communication channel is called a service provider. For example, Gupshup is a service provider for WhatsApp APIs, SMS APIs etc. Similarly there are other providers like Twilio, CDAC etc.
* **Adapter**: An adapter translates between messages received from communication channels on specific providers and the internal XMessage format (followed by UCI). A new adapter is required for every new combination of communication channel and service provider (e.g. - WhatsApp + Gupshup; WhatsApp + NetCore; WhatsApp + Twilio). An adapter config references the communication channel, service provider and associated metadata such as specific business account phone number.
* **Transformer**: A transformer is a stateless processing object that takes inputs and converts the input into a processed output. Transformers may in turn call external services if needed.
* **Conversation Logic**: Conversation logic defines the overall flow for a specific conversation. A conversation logic object references a sequence of transformers that will be applied to arrive at the final response at a specific point in the conversation, and the associated adapter config for this conversation logic.
* **User Segments :** A bot can either be an open public bot or a closed targetted bot. In the latter case, a closed user segment has to be defined. User segment contains user data including a mechanism to fetch them from a federated user registry. * **Conversations :** A conversation orchestrates a conversation with a specific conversation logic assigned to a set of users. A bot object references user segment(s) and conversation logic(s). # High-level Architecture Design The block diagram below outlines various components that come together to form the Unified Communications Interface (UCI): ![Tech Architecture Diagram](https://1242145269-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FnC2042spZ2TIbqdRA37S%2Fuploads%2Fp84UcDr9TpzHRXs2VpVX%2FUCI%20Architecture%20Updated.png?alt=media\&token=a85c7b55-ebfb-41a3-8f58-e78b001e5994) 1. **Adapter**: An adapter translates between messages received from communication channels on specific providers and the internal XMessage format (followed by UCI). A new adapter is required for every new combination of communication channel and service provider (e.g. - WhatsApp + Gupshup; WhatsApp + NetCore; WhatsApp + Twilio). An adapter config references the communication channel, service provider and associated metadata such as a specific business account phone number. 2. **Transformer**: A transformer is a stateless processing object that takes inputs and converts the input into a processed output. Transformers may in turn call external services if needed. 3. **Conversation Logic**: Conversation logic defines the overall flow for a specific conversation. A conversation logic object references a sequence of transformers that will be applied to arrive at the final response at a specific point in the conversation and the associated adapter config for this conversation logic. 4. **User Segments:** A bot can either be an open public bot or a closed targetted bot. In the latter case, a closed user segment has to be defined. User segment contains user data including a mechanism to fetch them from a federated user registry. 5. **Conversations:** A conversation orchestrates a conversation with a specific conversation logic assigned to a set of users. A bot object references user segment(s) and conversation logic(s). Following is a summary of the relation between the different blocks of UCI : `Adapter = Communication Channel + Service Provider` `User segment` `Transformer` `Conversation Logic = {Transformers} + Adapter` `Bot = {Conversation Logics} + {User segments}` | Real World | UCI Platform | | -------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------- | | Set of end users in a state | User segment 1 containing user registry uri, user ids, user metadata including phone numbers | | Single end user phone number part of the state (eg. 938490834) | User phone number field in user segment object | | Whatsapp | Communication Channel referenced in the Adapter 1 config | | Gupshup | Service provider referenced in the Adapter 1 config | | Business account phone number 908092348 | Meta data for adapter 1, referenced in Adapter config | | If fail, then this message | Transformer 1 referencing an XForm with pass/fail message logic | | Translate message from English to Hindi | Transformer 2 referencing a translation microservice | | Given fail, send message via Whatsapp to student | Conversation logic 1 = Transformer 1 + Transformer 2 + Adapter 1 | | Given fail, send same message via SMS to student | Conversation logic 2 = Transformer 1 + Transformer 2 + Adapter 2 (where adapter 2 config references SMS communication channel and SMS service provider) | | Lifecycle of conversation | Conversation = Conversation logic 1 + Conversation logic 2 + User segment 1 | # Design Principles **Plugin Based -** Adapters, Transformers and Users can all be injected by external providers. Allowing for un-opinionated implementation of specs (which are kept minimal) allowing for simpler, wider and faster adoption. **Minimal Micro-services -** A simple structure on how the micro-services will interact with each other (specs) and how the information will be stored and processed. ![](https://lh3.googleusercontent.com/OVWuTqk6shpts2Ar2fSX5Hui2_MGH5n2O7J_3JGtUF7LDs8iltWvW-FehjvpMhGE6L9XJkQeFWhThJW5P4KEiMESTIsdtD5Xmz-6fNGlbUCd9lXrBviDfhxa8It2sal-B9P0bhlr) **Federated Architecture** - We only save configuration as part of the UCI and everything is just called from respective places. ![](https://lh3.googleusercontent.com/IuaVEYpOVGECXdtJlY1Zg46Ub0b9z9aiEXZAmhrcLzWg5zk3NDhZI9zrfOzdIx4kIBhQr6zpztN3mH5I08B5vu-2TrO4K2Jr5QppIebWdel36KQZ58RX1kF0kl2HqqMnQZs_t2nz) **Data Federation** - All data is secured using JWT and with extreme automation on the building of APIs using data specs. ![](https://lh6.googleusercontent.com/kpq39YFklZIt1EuxVHMwTzRuqoLF5PKTWF8GIJOdo7yDAt5MdWLuGHLUgHXM9FSFrn9jD75kHJ5dS2fzu2QHDBFyzzT6FZM7oPbpMmq8sUbHv2RaVsosGh8_xMooy6W7lBQ5KBCC) **Population scale** - Asynchronous everywhere with independently scalable small services. Databases that scale horizontally - Cassandra for persistence; Kafka pipeline for decoupling the Saga transactions. Caching at all levels. Deployment auto scalable based on Kafka queue - HPA (k8s). # Standards 1. OpenWeb Standard such as - HTML, CSS, ECMAScript, LaTeX 2. Standard Content Format such as PDF, MP4, WebM, H5P, ePub, and more 3. XForm Specification 4. Accessibility - WCAG, Internationalisation 5. Security - PKI, HTTPS, JWT Most of these standards are underlying technological standards which are not applicable for auditing. UCI as an open-source tool can be extended to comply with or support more standards as necessary. # Choose your Persona Choose the Persona that fits you best to view next steps {% content-ref url="/pages/9gXAsnquMHihBHQjnKRf" %} [Adopter - Install and use UCI](/use/adopter) {% endcontent-ref %} {% content-ref url="/pages/yPdxRFIUQkA3aAFdOukU" %} [Developer - Contribute to UCI](/use/developer) {% endcontent-ref %} # Adopter - Install and use UCI I want to use Sunbird UCI in my own project and I want to set up my own instance of Sunbird UCI **Adoption guide - overview** This page contains guides which walk through various aspects of UCI functionality. These documents are meant to give you a thorough understanding of the technology covered. [**Pre-requisites, Installation Setup, Post Setup**](/use/adopter/pre-requisites-installation-setup-post-setup) This section lists everything you need to know about setting up the UCI on your own server. It starts with the infra pre-requisites, then details the Docker-based installation setup. At the end, the section covers the steps to test out various services that should be up. If you face any problem with one or more services, you can reach out to the community for help [here](https://github.com/samagra-comms/community/discussions). [**Setting up your very first conversation**](/use/adopter/setting-up-your-very-first-conversation) After the setup is complete. The next step is to create a conversation and to start testing the UCI for an end to end custom logical conversation flow. There are two ways listed on this page: 1. Using the admin console to create the bot 2. Using the APIs in the sequence to create the bot\ *The admin console internally uses the same APIs calls as mentioned in the second method* After the bot is created you can actually test the bot on the UCI web channel. Web channel is an in-house created channel that can be used standalone or can be embedded in any existing application. It does not require any separate setup. It comes as a package with vanilla UCI setup. [**API Documentation**](/use/adopter/api-documentation) This section lists all the relevant APIs required to use the UCI services. More APIs can be found in the Developer section of the documentation. [**Data Exhaust and Analytics**](/use/adopter/data-exhaust-and-analytics) This section captures how the usage data for the bot can be extracted and used further for analytics purposes. [**PostHog events**](/use/adopter/posthog-event) UCI does not differentiate between any data storage and analytics tool. We are talking about PostHog as we found it to fit the bill for our needs. In this section, we share how you can setup Posthog, view events on PostHog and also create your relevant dashboards on it. # Pre-requisites, Installation Setup, Post Setup ## Pre-requisites, Installation Setup, Post Setup [Pre-requisites](#pre-requisites) [Installation](#installation-setup) [Post Setup](#post-setup) ## Pre-requisites 1. Install Docker if not already installed. [Click Here](https://docs.docker.com/engine/install/ubuntu/) 2. Infra : * 8 CPU Cores * 16 GB RAM * 20 GB Storage * Server Fixed/Static IP * Preferred OS - Linux 3. Basic familiarity is required with: * Terminal & its commands * Docker * Server * Rest apis 4. Please make sure all of the ports mentioned used in the [file](https://github.com/samagra-comms/docker-deploy/blob/main/docs/ports.md) are open & are not being used by any other service on the server. ## Installation Setup 1. Take clone of this repository. `git clone https://github.com/samagra-comms/docker-deploy.git` 2. Go to folder `cd docker-deploy` 3. Install the requirements `pip3 install -r requirements.txt` 4. Fill in the environment variables in the `.env` file. Find out more about the `.env` [here](https://uci.sunbird.org/use/adopter/about-the-env). 5. Run below command to download & start the services using docker. `python3 install.py {ENCRYPTION_KEY} {NETCORE_WHATSAPP_AUTH_TOKEN} {NETCORE_WHATSAPP_SOURCE} {NETCORE_WHATSAPP_URI}` The encryption key should be a strong string consisting uppercase, lowercase, numeric and special characters. For setting up a local demo, it can be anything. For example: 1234. The rest of the parameteres are optional, they are only required for the whatsapp flow. If you don't want to test it out, then you are good to go only with the encryption key. In that case, the final command comes out to be: `python3 install.py 1234` * If script fails on runtime ,run the following commands and re-run the script * Stop the all services and volumes `docker-compose down -v` * Remove the data directory `sudo rm -rf data` * Run the script again 6. Follow the instructions in the script. At one point, the script will ask you to run some queries in the hasura console to fill in adapters. Follow the instructions there to do so. 7. Once the setup is completed, go to the web channel () to test out the conversation with the bot. **Note**: Please note this installation is based on a very standard and generalized use case. If your needs are not fulfilled with the current installation, please start scaling the individual services by using them in docker stack. #### **After a Successful Docker Setup Completion** 1. Script execution time - **Around 30 minutes** (depends on your server) 2. On a successful run of the script you will get below services: * Inbound Service: * Orchestrator Service: * Transformer Service: * Outbound Service: * Broadcast Transformer Service: * Campaign Service: * Kafka UI: * Bot DB Hasura UI: * Fusion Auth: * ODK: * Chat Frontend: * Admin Console: **Note**: If above mentioned url's are not working then restart the all docker services or check the docker container logs. 3. If you want to check logs for a specific service, follow below flow. * Show all docker containers `docker-compose logs {service_name} --follow` **Note**: If the services are updated on any server, use server's ip instead of localhost. Eg. ## **Post Setup** 1. You can start using FusionAuth Console using and create an Account, for managing users and what resources they are authorized to access. 2. For managing all the assesment data go on URL : to view all the submissions **Note**: If you face any difficulty in setting up UCI, please raise an issue [here](https://github.com/samagra-comms/docker-deploy/issues). All the issues that you raise will help us make UCI make better for adopters. # Setting up your very first conversation ## 1. **Setting up your first bot** ### 1.1. UCI Admin * Go to admin console. You can find the repo [here](https://github.com/samagra-comms/uci-admin). * Click on the "Add new button". * Fill the form with a unique starting message, start date equals to current & end date more than the current date. * Click on the Next button. * Click on Add logic button. * Fill the form & upload a xml form. * Add this & submit this form. * The bot will be added and we can start using this on the UCI front. ### 1.2. APIs #### 1.2.1 Upload ODK Form * Convert a ODK Excel form to XML form using [Link](https://getodk.org/xlsform/). [Sample ODK Excel Form](https://github.com/samagra-comms/docker-deploy/blob/main/media/List-QRB-Test-Bot.xlsx) * Upload this XML from using this api. ``` curl --location 'http://localhost:9999/admin/form/upload' \ --header 'admin-token: {ADMIN_TOKEN}' \ --form 'form=@"{PATH_TO_YOUR_XML}"' \ --form 'mediaFiles=@"{PATH_TO_YOUR_MEDIA_1}"' \ --form 'mediaFiles=@"{PATH_TO_YOUR_MEDIA_2}"' \ --form 'mediaFiles=@"{PATH_TO_YOUR_MEDIA_N}"' ``` [Sample ODK XML Form](https://github.com/samagra-comms/docker-deploy/blob/main/List-QRB-Test-Bot.xml) **Response**: The API will return a form id. Use this form id to create conversation logic API. Form id E.g. **testing\_form** ``` { "apiId": "api.form.upload", "path": "/admin/form/upload", "msgid": "fec3421b-cc04-42e2-9ce9-e004b3ded827", "result": { "status": "UPLOADED", "data": { "formID": "testing_form" } }, "startTime": "2023-07-04T12:01:00.268Z", "method": "POST", "endTime": "2023-07-04T12:01:21.785Z" } ``` #### 1.2.2 Create a Conversation Logic ``` curl --location 'http://localhost:9999/admin/conversationLogic' \ --header 'admin-token: {ADMIN_TOKEN}' \ --header 'Content-Type: application/json' \ --header 'Cookie: fusionauth.locale=en_US; fusionauth.sso=AgOat0GjncGOHhPpH_HuL9QQqnfMitd15O-ofS-uTcdA' \ --data '{ "data": { "id": null, "name": "UCI List & Button Logic", "description": "UCI List & Button Logic Desc", "transformers": [ { "id": "bbf56981-b8c9-40e9-8067-468c2c753659", "meta": { "form": "https://hosted.my.form.here.com", "formID": "List-Button-test-v1" } } ], "adapter": "44a9df72-3d7a-4ece-94c5-98cf26307323" } }' ``` **Response**: It will return a conversation logic id, use it in create bot. Eg. **92f7b965-4118-4ddc-9c7d-0bc0f77092db** ``` { "ts": "2022-05-24T13:48:06.407Z", "params": { "resmsgid": "23b94970-db68-11ec-ae84-fbd67a9c1174", "msgid": null, "status": "successful", "err": null, "errmsg": null }, "responseCode": "OK", "result": { "data": { "transformers": "[{"id":"bbf56981-b8c9-40e9-8067-468c2c753659","meta":{"form":"https://hosted.my.form.here.com/%22,/%22formID/%22:/%22List-Button-test-v1/%22%7D%7D}}]", "adapter": "44a9df72-3d7a-4ece-94c5-98cf26307323", "name": "UCI demo bot logic", "id": "92f7b965-4118-4ddc-9c7d-0bc0f77092db" } } } ``` #### 1.2.3 Create a bot ``` curl --location 'http://http//localhost:3002/admin/bot' \ --header 'asset: bot' \ --header 'admin-token: dR67yAkMAqW5P9xk6DDJnfn6KbD4EJFVpmPEjuZMq44jJGcj65' \ --header 'Accept: application/json, text/plain, */*' \ --header 'ownerOrgID: org01' \ --header 'ownerID: 8f7ee860-0163-4229-9d2a-01cef53145ba' \ --form 'botImage=@"{PATH_TO_IMAGE}"' \ --form 'data="{ \"data\": { \"name\": \"TEST ODK BOT\", \"description\": \"TEST\", \"purpose\": \"TEST\", \"startingMessage\": \"Hi Test ODK\", \"startDate\": \"2023-06-16\", \"endDate\": \"2023-06-30\", \"isBroadcastBotEnabled\": true, \"segmentId\": \"1\", \"status\": \"enabled\", \"users\": [], \"logic\": [ \"92f7b965-4118-4ddc-9c7d-0bc0f77092db\" // Get this id from conversation logic api response ] } }"' ``` **Response**: This api will return a bot id & other bot information. Use the starting message (Eg. **Hi Test ODK**) from here to start conversation with a bot. ``` { "apiId": "api.admin.bot", "path": "/admin/bot", "apiVersion": "v1", "msgid": "ba858dbb-8710-40e3-9fca-e8f928ff2cfc", "result": { "id": "5066c217-6ae0-4e3c-9802-8c0cce660c2b", "createdAt": "2023-07-04T12:59:48.804Z", "updatedAt": "2023-07-04T12:59:48.804Z", "name": "TEST ODK BOT", "startingMessage": "Hi Test ODK", "ownerID": null, "ownerOrgID": null, "purpose": null, "description": null, "startDate": "2023-06-15T00:00:00.000Z", "endDate": "2025-12-01T00:00:00.000Z", "status": "ENABLED", "tags": [], "botImage": "1c8516c6-c4c3-4a37-a8dd-89109e8e0e60.png" }, "startTime": "2023-07-04T12:59:48.265Z", "method": "POST", "endTime": "2023-07-04T12:59:49.097Z" } ``` **Note**: If you want to use the gupshup adapter, please contact the [administrator](https://github.com/samagra-comms/docker-deploy#contact-administrator) for credentials. ## **2. Start Using Bots** ### Using Web Channel Once the bot is created, we can start using it. Send the starting message added in the [**Create a bot**](#1.2.3-create-a-bot) api. **Example :** Hi Test Bot ![](https://1242145269-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FnC2042spZ2TIbqdRA37S%2Fuploads%2FzKQAHLcdvIuzlhRfULee%2FTest-Bot-Flow-pwa.png?alt=media\&token=8d9db6b9-72e1-4ff8-95b9-024fe88aa3ec) ### Using Whatsapp Once the bot is created, we can start using it. If you have set up gupshup/netcore provider for whatsapp, Send the starting message added in the [**Create a bot**](#1.2.3-create-a-bot) api to the whatsapp number. ![](https://1242145269-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FnC2042spZ2TIbqdRA37S%2Fuploads%2Ff7In8dEPYwBNws1D9LGQ%2FTest-Bot-Flow-Whatsapp.jpeg?alt=media\&token=201e61c4-998d-4ddd-bfe4-828f1bcb65c2) # API Documentation **I want to use UCI for my own use case how do I proceed?** 1. Setup UCI on your infra - [Steps](/use/adopter/pre-requisites-installation-setup-post-setup) 2. Is the channel and provider you want to use is already supported in UCI? 1. **If yes** -> Follow [these steps](https://uci.sunbird.org/use/adopter/pages/iyh1Cgn2i7ue92G4ym2v#2.1-add-adapter) to add your adapter along with your own credentials for the platform 2. **If No** -> Follow [these steps](https://uci.sunbird.org/use/adopter/pages/3IrzE78Ty9mUiLsrjHGd#3.-creating-your-own-adapters) to contribute a new adapter and then follow the step above 3. Follow these steps to [create your own bot](/use/adopter/setting-up-your-very-first-conversation) **In my use case, I want to send direct notifications to the users, I do not want to create a complete conversation flow. How Can I achieve this?** 1. Setup UCI on your infra - [Steps](/use/adopter/pre-requisites-installation-setup-post-setup) 2. Is the channel and provider you want to use is already supported in UCI? 1. **If yes** -> Follow [these steps](https://uci.sunbird.org/use/adopter/pages/iyh1Cgn2i7ue92G4ym2v#2.1-add-adapter) to add your adapter along with your own credentials for the platform 2. **If No** -> Follow [these steps](https://uci.sunbird.org/use/adopter/pages/3IrzE78Ty9mUiLsrjHGd#3.-creating-your-own-adapters) to contribute a new adapter and then follow the step above 3. Follow the [direct Message APIs](/use/developer/api-documentation/direct-message-apis) to send the messages directly # Data Exhaust and Analytics ### **Overview** We can generate a data exhaust, which is a list of conversations between the bot and the user. When the bot and user converse, our service sends an event (AKA telemetry event) to a third-party service, which then uploads it to the blob storage. When a message is sent to the user and the user replies to it, a telemetry event will be generated and will be sent to the third-party service. We can then download a list of these events for a specific conversation for a specific time period using the APIs mentioned in the document. The telemetry event sent is generated according to the specification mentioned in the [link](https://github.com/sunbird-specs/Telemetry/blob/master/learn/specification.md). ### **Data Exhaust Requests** We can generate a job for two types of exhaust requests. **1. UCI Response Exhaust** This will provide a list of conversations done by any user between the time period mentioned in the exhaust request. Each event consists of the user device id, question asked, question options if any, question reply from the user, timestamp, etc. A conversation event is basically a telemetry event that we send for each conversation message. A sample for the [UCI response exhaust](https://github.com/samagra-comms/community/blob/main/media/uci-response-exhaust.csv) **2. UCI Private Exhaust** This will provide a list of users that have conversed with the bot. It consists of user phone number(If consented by user), device id, username etc. The device id is the one that connects the response & private exhaust. To identify the phone number for the conversation done by the user, check the device id from the response exhaust, it should exists in the private exhaust. A sample for the [UCI private exhaust](https://github.com/samagra-comms/community/blob/main/media/uci-private-exhaust.csv) ### Data Exhaust Apis #### **1. API to submit a job request for response/private exhaust** This API will submit a job request to fetch the response/private exhaust for a specific conversation(bot) id based on the dataset config. The API will give a requestID in the response. The requestID & tag are used to fetch the job info. * URL: ``` {{host}}/dataset/v1/request/submi ``` * Method: `POST` * Sample Request Body: ``` { "id": "ekstep.analytics.job.request.submit", "ver": "1.0", "ts": "2022-02-15T10:07:00+05:30", "params": { "msgid": "06fb2660-88ba-11ec-bea0-578c52d1ac65" }, "request": { "dataset": "uci-response-exhaust", // dataset should be "uci-private-exhaust" or "uci-response-exhaust" "tag": "7ed871f7-0d5b-4bb0-86a0-a00151352706", // random UUID "requestedBy":"user_id", "requestedChannel": "01309282781705830427", // channel id for the bot "datasetConfig": { "type": "uci-response-exhaust", // type should be "uci-private-exhaust" or "uci-response-exhaust" "conversationId": "c63b6faa-11a9-4259-b941-11cd2f8446c6", // bot id "startDate": "2022-03-01", // fetch the response data from date "endDate": "2022-04-01" // fetch the response data to date }, "output_format":"csv" //output format } } ``` * Sample Response ``` { "id": "ekstep.analytics.dataset.request.submit", "ver": "1.0", "ts": "2022-04-06T09:32:00.079+00:00", "params": { "resmsgid": "d15de3c9-875d-4fbd-912c-4daa1cee0f1b", "status": "successful", "client_key": null }, "responseCode": "OK", "result": { "attempts": 0, "lastUpdated": 1649237520065, "tag": "7ed871f7-0d5b-4bb0-86a0-a00151352706:01269878797503692810", // Tag to be used in job info api, but without the channel id "expiresAt": 1649239320077, "datasetConfig": { "type": "uci-response-exhaust", "conversationId": "c63b6faa-11a9-4259-b941-11cd2f8446c6", "startDate": "2022-03-01", "endDate": "2022-04-01" }, "downloadUrls": [], "requestedBy": "user_id", "jobStats": { "dtJobSubmitted": 1649237520065, "dtJobCompleted": null, "executionTime": null }, "status": "SUBMITTED", "dataset": "uci-response-exhaust", "requestId": "1E2A13DA30671FDBA10C418C9EF14A70", // Request id to be used in job info api "requestedChannel": "01269878797503692810" } } ``` #### **2. API to get job info of a job** This API is used to get the job info which includes the job status (Eg. Submitted, Success, Failed), download URLs, (if the job is successful), job request param etc. Tag & Request Id for the request will be used from the submitted job request. The download url is a signed url which is valid for one day only. * URL ``` {{host}}/dataset/v1/request/read/{{tag(bot/conversation Id)}}?requestId={{requestId}} ``` ``` Eg: {{host}}/dataset/v1/request/read/7ed871f7-0d5b-4bb0-86a0-a00151352703?requestId=1E2A13DA30671FDBA10C418C9EF14A70 ``` * Method: `GET` * Sample Request Params ``` requestId=1E2A13DA30671FDBA10C418C9EF14A70 ``` * Sample Response ``` { "id": "ekstep.analytics.dataset.request.info", "ver": "1.0", "ts": "2022-04-06T09:36:21.181+00:00", "params": { "resmsgid": "f5c95bc2-f643-41f9-adb1-d566d03cf411", "status": "successful", "client_key": null }, "responseCode": "OK", "result": { "attempts": 0, "lastUpdated": 1649165260152, "tag": "7ed871f7-0d5b-4bb0-86a0-a00151352703:01269878797503692810", "expiresAt": 1649239580993, "datasetConfig": { "type": "uci-response-exhaust", "conversationId": "c63b6faa-11a9-4259-b941-11cd2f8446c6", "startDate": "2022-03-01", "endDate": "2022-04-01" }, "downloadUrls": [ "https://sunbirddevprivate.blob.core.windows.net/reports/uci-response-exhaust/1E2A13DA30671FDBA10C418C9EF14A70/c63b6faa-11a9-4259-b941-11cd2f8446c6_response_20220405.csv?sv=2017-04-17&se=2022-04-06T10%3A06%3A21Z&sr=b&sp=r&sig=L5etefAbBA/2PUQHaqSvbDPbyC%2BPfftS2jkf8y64R6E%3D" // download url for the request exhaust ], "requestedBy": "user_id", "jobStats": { "dtJobSubmitted": 1649165118347, "dtJobCompleted": 1649165260152, "executionTime": 6210 }, "status": "SUCCESS", "dataset": "uci-response-exhaust", "requestId": "1E2A13DA30671FDBA10C418C9EF14A70", "requestedChannel": "01269878797503692810", "statusMessage": "" } } ``` #### **3. API to get joblist** This API is used to fetch the jobs list for a specific tag/bot. * URL ``` {{host}}/dataset/v1/request/list/{{tag(bot/conversation Id)}}?limit=5 ``` ``` Eg. {{host}}/dataset/v1/request/list/7ed871f7-0d5b-4bb0-86a0-a00151352703?limit=5 ``` * Method: `GET` * Sample Response ``` { "id": "ekstep.analytics.dataset.request.list", "ver": "1.0", "ts": "2022-04-06T09:38:26.540+00:00", "params": { "resmsgid": "5aebb821-b821-4c66-a5da-1df3af84da65", "status": "successful", "client_key": null }, "responseCode": "OK", "result": { "count": 1, "jobs": [ { "requestId": "84B26F9012D6E2E53BFC422B245C6350", "tag": "7ed871f7-0d5b-4bb0-86a0-a00151352703:ORG_001", "dataset": "uci-response-exhaust", "requestedBy": "user_id", "requestedChannel": "ORG_001", "status": "SUCCESS", "lastUpdated": 1649165260152, "datasetConfig": { "type": "uci-response-exhaust", "conversationId": "d655cf03-1f6f-4510-acf6-d3f51b488a5e", "startDate": "2022-03-01", "endDate": "2022-04-01" }, "attempts": 0, "jobStats": { "dtJobSubmitted": 1649165118347, "dtJobCompleted": 1649165260152, "executionTime": 6210 }, "downloadUrls": [ "https://sunbirddevprivate.blob.core.windows.net/reports/uci-response-exhaust/84B26F9012D6E2E53BFC422B245C6350/d655cf03-1f6f-4510-acf6-d3f51b488a5e_response_20220405.csv?sv=2017-04-17&se=2022-04-06T10%3A08%3A26Z&sr=b&sp=r&sig=6N9Kz4P/t9wZ1sCn3mYdVWZ22IfyyqrUNp3dNyelpJ0%3D" ], "expiresAt": 1649239706533, "statusMessage": "" } ] } } ``` Postman collection: [Link](https://github.com/samagra-comms/community/blob/main/media/DataExhaust-APIs-postman_collection.json) Environment: [Link](https://github.com/samagra-comms/community/blob/main/media/Data%20Exhuast-postman_environment.json) ### Execute Data Exhaust Job The jobs will be executed via a cron job that runs daily at 02.00 AM. We should check the job info, the next day to check the job's current status. If we need an immediate result we can also run a job from the Jenkins AnalyticsReplayJobs service. Follow the below step to execute the jobs. * Login to Jenkins & Go to the AnalyticsReplayJobs service from the Deploy->Dev->DataPipeline services list or directly hit the [url](https://10.20.0.14/jenkins/job/Deploy/job/dev/job/DataPipeline/job/AnalyticsReplayJobs/). * Go to Build with Parameter. * Select `run-job` from the `job_type` drop down. * Select `uci-response-exhaust`/`uci-private-exhaust` from the `job_id` drop down. [![Screenshot](https://github.com/samagra-comms/community/raw/main/use/media/exhaust-jenkins-job.png)](https://github.com/samagra-comms/community/blob/main/use/media/exhaust-jenkins-job.png) * Leave the rest as it is & click build. It will run all the submitted jobs. * When it's done, run the [job info api](#2.-api-to-get-job-info-of-a-job) to fetch the job response. The job will have either a success/failed status. If the job is successful it will have a download url in the response. ``` Eg. https://sunbirddevprivate.blob.core.windows.net/reports/uci-response-exhaust/84B26F9012D6E2E53BFC422B245C6350/d655cf03-1f6f-4510-acf6-d3f51b488a5e_response_20220405.csv?sv=2017-04-17&se=2022-04-06T10%3A08%3A26Z&sr=b&sp=r&sig=6N9Kz4P/t9wZ1sCn3mYdVWZ22IfyyqrUNp3dNyelpJ0%3D ``` # Posthog Event ## Overview PostHog is the all-in-one product analytics suite. PostHog provides everything product-led teams need in one place, including: * Product Analytics * Aggregate Dashboard * Insights * User lifecycle * Event pipelines * Export data In UCI we are also using PostHog for creating aggregate dashboard by sending the events on each conversation. The events follow given [specification](https://github.com/sunbird-specs/Telemetry/blob/master/learn/specification.md).
## Steps to use PostHog events ### 1. Create Posthog Account First we would need to create a new account on [Posthog](https://app.posthog.com/signup) & get a project api key. The project api key will be available in [Project Settings](https://app.posthog.com/project/settings) of Posthog account. ### 2. Enable Posthog Event in UCI To enable posthog events in UCI follow below steps. 1. Add below environment variables for posthog url & api key. [Link to environments of UCI](/use/developer/development-environment).
``` POSTHOG_TELEMETRY_APIKEY= // Api key POSTHOG_TELEMETRY_URL=https://app.posthog.com ``` 2. Add environment to enable posthog event.
``` POSTHOG_EVENT_ENABLED=TRUE ``` 3. After the posthog events are enabled, UCI will send all conversations as events to posthog. we can generate aggregate dashboard with these events using insights, actions etc. ### 3. Create insights in Posthog We can create insights like counts, graphs etc using posthog insights. Use this [documentation](https://posthog.com/docs) to better understand the insights, funnels, life-cycle etc that posthog provides. # Whatsapp (Netcore) ## Environment Variables Required * NETCORE\_WHATSAPP\_AUTH\_TOKEN * NETCORE\_WHATSAPP\_SOURCE * NETCORE\_WHATSAPP\_URI In order to test out the whatsapp flow, you need to provide these to the installation script while setting up UCI. Refer to the docs [here](https://uci.sunbird.org/use/adopter/pre-requisites-installation-setup-post-setup) ## Types of Bots: 1. **Broadcast Bot:** This type of bot is used to send a single message to a large group of people. It doesn't expect a reply to the message. 2. **Conversation Bot:** This type of bot is designed for a two way conversation with a user. The conversation is regulated by an ODK form. ## Broadcast Flow This is a 4 step process. 1. Create conversation logic: ```bash curl --location 'http://localhost:3002/admin/conversationLogic' \ --header 'admin-token: 4737ea8520bd454caabb7cb3d36e14bc1832c0d3f70a4189b82598670f11b1bd' \ --header 'Content-Type: application/json' \ --header 'Cookie: fusionauth.locale=en_US; fusionauth.sso=AgOat0GjncGOHhPpH_HuL9QQqnfMitd15O-ofS-uTcdA' \ --data '{ "data": { "id": null, "name": "Netcore Demo", "description": "Netcore Demo", "transformers": [ { "id": "774cd134-6657-4688-85f6-6338e2323dde", "meta": { "body": "Hello ${name}-${phoneNo}, Netcore Test Notification Body demo dry run.", "type": "broadcast", "title": "Netcore Demo", "params": [ "name", "phoneNo" ], "templateType": "JS_TEMPLATE_LITERALS" } } ], "adapter": "44a9df72-3d7a-4ece-94c5-98cf26307323" } }' ``` | Parameter | Description | | ------------ | ----------------------------------------------------------------------------- | | id | The ID of the conversation logic. | | name | The name of the conversation logic. | | description | The description of the conversation logic. | | transformers | An array of transformer objects that define the message content and behavior. | | | - id: The ID of the transformer. | | | - meta: An object containing metadata about the transformer. | | | - body: The message content. | | | - type: The type of message, e.g., "broadcast". | | | - title: The title of the message. | | | - params: An array of parameter names used in the message template. | | | - templateType: The template type, e.g., "JS\_TEMPLATE\_LITERALS". | | adapter | The ID of the adapter associated with the conversation logic. | This will create a conversation logic for the broadcast flow. We don't need an ODK form here as this is not a two-way conversation flow that is driven by ODK logic. The conversation logic for this type of bot only has a single broadcast message which is defined in the meta of the request. 2. Create user segment: Creating a user segment is analogous to importing a list of users into UCI. The curl for this is given below. ```bash curl --location 'http://localhost:3002/admin/user-segment' \ --header 'admin-token: 4737ea8520bd454caabb7cb3d36e14bc1832c0d3f70a4189b82598670f11b1bd' \ --header 'Content-Type: application/json' \ --data '{ "name": "Netcore demo", "all": { "type": "get", "config": { "url": "http://localhost:9080/service/testUserSegment2", "type": "GET", "cadence": { "perPage": 1, "retries": 5, "timeout": 60, "concurrent": true, "pagination": true, "concurrency": 10, "retries-interval": 10 }, "pageParam": "page", "credentials": {}, "totalRecords": 1 } }, "byID": { "type": "get", "config": { "url": "http://localhost:9080/service/testUserSegment2", "type": "GET", "cadence": { "perPage": 1, "retries": 5, "timeout": 60, "concurrent": true, "pagination": true, "concurrency": 10, "retries-interval": 10 }, "pageParam": "page", "credentials": {}, "totalRecords": 1 } }, "byPhone": { "type": "get", "config": { "url": "http://localhost:9080/service/testUserSegment2", "type": "GET", "cadence": { "perPage": 1, "retries": 5, "timeout": 60, "concurrent": true, "pagination": true, "concurrency": 10, "retries-interval": 10 }, "pageParam": "page", "credentials": {}, "totalRecords": 1 } }, "count": 0 }' ``` This creates a new user segment with the user list provided on `http://localhost:9080/service/testUserSegment2`. | Parameter | Description | | --------- | ------------------------------------------------------------------------------------------- | | name | The name of the user segment. | | all | Configuration for retrieving all users in the segment. | | | - type: Type of HTTP request (e.g., "get") for retrieving users. | | | - config: An object containing configuration settings for the HTTP request. | | | - url: The URL to which the HTTP request will be sent to retrieve user data. | | | - type: The type of HTTP request (e.g., "GET"). | | | - cadence: An object containing settings related to timing and concurrency of the requests. | | | - pageParam: The parameter name used for pagination in the URL's query parameters. | | | - credentials: An object containing credentials (if required) for making the request. | | | - totalRecords: A field indicating the total number of records in the user segment. | | byID | Configuration for retrieving users by their IDs. | | byPhone | Configuration for retrieving users by their phone numbers. | | count | A field indicating the count of users in the user segment. | The user list on the provided URL should be in the following format: ```json { "data": [ { "phoneNo": "", "name": "", } ] } ``` | Parameter | Description | | ------------ | ----------------------------- | | data.phoneNo | The phone number of the user. | | data.name | The name of the user. | These params are necessary for the flow to work. Apart from these params, any additional params which are defined while creating conversation logic should also be provided in this object. ### Note: This user list definition is only valid for Whatsapp flow. Definitions will differ for different use cases. 3. Create a broadcast bot: ```bash curl --location 'http://localhost:3002/admin/bot' \ --header 'admin-token: 4737ea8520bd454caabb7cb3d36e14bc1832c0d3f70a4189b82598670f11b1bd' \ --form 'botImage=@"/home/ryanwalker277/Pictures/Screenshot from 2023-08-08 15-10-09.png"' \ --form 'data="{ \"data\": { \"startingMessage\": \"Hi Netcore demo\", \"name\": \"Hi Netcore demo\", \"users\": [\"1be4e149-3618-4135-84ac-f39c49d6f671\"], \"logic\": [ \"2b16fd7b-6c87-4726-9035-c20400901a82\" ], \"status\": \"enabled\", \"startDate\": \"2023-03-01\", \"endDate\": \"2024-12-01\", \"ownerid\":\"8f7ee860-0163-4229-9d2a-01cef53145ba\", \"ownerorgid\":\"org01\", \"purpose\":\"Testing\", \"desc\":\"Netcore Broadcast Bot Desc\" } }"' ``` | Parameter | Description | | -------------------- | -------------------------------------------------------------------- | | botImage | The image file of the bot to be uploaded. | | data | A JSON object containing various configuration settings for the bot. | | data.startingMessage | The initial message of the bot to trigger the conversation flow. | | data.name | The name of the bot. | | data.users | An array of user segments attached to the bot | | data.logic | Conversation logic associated with the bot. | | data.status | The status of the bot ("enabled" or "disabled"). | | data.startDate | The start date of the bot's activity. | | data.endDate | The end date of the bot's activity. | | data.ownerid | The ID of the bot's owner. (Vault) | | data.ownerorgid | The organization ID of the bot's owner. (Vault) | | data.purpose | A description of the purpose of the bot. | | data.desc | A description of the bot. | At the time of bot creation, it must be attached to a user segment. This is done by passing the segment id, which is a unique identifier for the user segement created in the previous step, in the users section of the request body. The segment id can be obtained from the response of the previous step. Add the conversation logic id to the body in logic section. 4. Triggering the bot: ```bash curl --location 'http://localhost:3002/campaign/start?campaignId=859ed55f-c276-4f38-ba72-5f08e1733705' ``` Replace the campaignId with the bot id given by the bot creation api.This will trigger a broadcast notification for all the users in that segment over whatsapp. ## Conversation Flow: This is a 3 step process. 1. Upload ODK form: ```bash curl --location 'http://localhost:3002/admin/form/upload' \ --header 'ownerId: 8f7ee860-0163-4229-9d2a-01cef53145ba' \ --header 'ownerOrgId: org1' \ --header 'admin-token: 4737ea8520bd454caabb7cb3d36e14bc1832c0d3f70a4189b82598670f11b1bd' \ --form 'form=@"/home/ryanwalker277/Music/WhatsappDemo/SampleOdkForm.xml"' ``` | Parameter | Description | | ----------- | ----------------------------------------------------- | | ownerID | The ID of the owner of the form. (Vault) | | ownerOrgId | The organization ID of the owner of the form. (Vault) | | admin-token | The admin token for authentication. | | form | The XML file of the form to be uploaded. | Refrence to create an [ODK Form](https://docs.getodk.org/xlsform/) In the conversation logic, ODK form becomes necessary as the whole conversation flow is determined by this. 2. Create conversation logic: ```bash curl --location 'http://localhost:3002/admin/conversationLogic' \ --header 'admin-token: 4737ea8520bd454caabb7cb3d36e14bc1832c0d3f70a4189b82598670f11b1bd' \ --header 'Content-Type: application/json' \ --header 'Cookie: fusionauth.locale=en_US; fusionauth.sso=AgOat0GjncGOHhPpH_HuL9QQqnfMitd15O-ofS-uTcdA' \ --data '{ "data": { "id": null, "name": "Netcore Demo", "description": "Necore Desc", "transformers": [ { "id": "bbf56981-b8c9-40e9-8067-468c2c753659", "meta": { "form": "https://hosted.my.form.here.com", "formID": "List-Button-test-v5" } } ], "adapter": "44a9df72-3d7a-4ece-94c5-98cf26307323" } }' ``` | Parameter | Description | | ----------------------------- | ---------------------------------------------------- | | data | An object containing the conversation logic data. | | data.id | The ID of the conversation logic (null for new). | | data.name | The name of the conversation logic. | | data.description | The description of the conversation logic. | | data.transformers.id | The ID of the transformer. | | data.transformers.meta | An object containing metadata about the transformer. | | data.transformers.meta.form | The URL of the form associated with the transformer. | | data.transformers.meta.formID | The ID of the form associated with the transformer. | | data.adapter | The ID of the adapter associated with the logic. | Replace the formId with the id of the form uploaded. This returns a unique conversation logic id. 3. Create conversation bot: ```bash curl --location 'http://localhost:3002/admin/bot' \ --header 'admin-token: 4737ea8520bd454caabb7cb3d36e14bc1832c0d3f70a4189b82598670f11b1bd' \ --form 'botImage=@"/home/ryanwalker277/Music/WhatsappDemo/boticon.jpeg"' \ --form 'data="{ \"data\": { \"startingMessage\": \"Hi uci\", \"name\": \"Netcore Demo\" , \"users\": [], \"logic\": [\"38dde1ef-d618-4440-adc7-34a7df885187\"], \"status\": \"enabled\", \"startDate\": \"2023-05-4\", \"endDate\": \"2025-12-01\" } }"' ``` | Parameter | Description | | -------------------- | ------------------------------------------------- | | botImage | The image file of the bot to be uploaded. | | data | An object containing the bot data. | | data.startingMessage | The initial message of the bot. | | data.name | The name of the bot. | | data.users | An array of user IDs associated with the bot. | | data.logic | Conversation logic associated with the bot. | | data.status | The status of the bot ("enabled" or "disabled"). | | data.startDate | The start date of the bot's activity. | | data.endDate | The end date of the bot's activity. | | botImage | The image file to be uploaded as the bot's image. | Notice that we are not passing the segment id here. That means this bot is not attached to any user segment. So, any user can interact with this bot by sending the starting message. # Environment Variables ### Environment Variables Explanation ### 1. `POSTHOG_API_KEY` The `POSTHOG_API_KEY` environment variable is used for authentication with the PostHog service. PostHog is a product analytics platform that helps you track user interactions with your application. To obtain this API key: 1. Log in to your PostHog dashboard. 2. Navigate to the API settings or account settings section. 3. Generate or retrieve your API key. 4. Set the `POSTHOG_API_KEY` environment variable to the obtained key. ### 2. `FIREBASE_SERVER_KEY` The `FIREBASE_SERVER_KEY` environment variable is typically used for Firebase Cloud Messaging (FCM) to send push notifications to mobile devices. To obtain this key: 1. Log in to your Firebase Console. 2. Create or select a project. 3. Navigate to the project settings. 4. Under the Cloud Messaging tab, find the "Server Key" or "Legacy Server Key." 5. Set the `FIREBASE_SERVER_KEY` environment variable to this key. ### 3. `NL Depenencies` NL app is our prime use case for UCI. UCI uses NL services for various tasks, priamrary of which is user authentication. Refer to [this](https://github.com/Mission-Prerna/sandbox-deployment) if you want to setup NL. We have used the [NL Sandbox](https://sandbox.bot.nl.samagra.io) for initial setup purpose. #### NLAPP\_USER\_URL * URL for an NL app dependency that searches for users by query. #### NLAPP\_USER\_AUTH * Authentication token for accessing the NL app user search API. #### NLAPP\_USER\_XAPPID * X-App-Id for the NL app. #### NG\_APP\_url * URL for an NG app. #### NG\_APP\_nl\_url * URL for the NL component of the NG app. #### NG\_APP\_user\_segment\_url * URL for user segmentation in the NG app. #### NG\_APP\_nl\_login\_url * URL for NL login in the NG app. #### NG\_APP\_nl\_login\_token * Token for NL login in the NG app. #### NG\_APP\_nl\_application\_id * Application ID for the NL app in NG. #### NG\_APP\_blobUrl * Blob URL for the NG app. #### NG\_APP\_botPhoneNumber * Phone number for the bot in the NG app. #### NG\_APP\_adapterId * Adapter ID for the NG app. #### NG\_APP\_broadcastAdapterId * Broadcast adapter ID for the NG app. #### NG\_APP\_userId * User ID for the NG app. #### NG\_APP\_orgId * Organization ID for the NG app. #### NG\_APP\_token * Token for the NG app. ### 4. Minio CDN Configuration: The following environment variables are used to configure Minio CDN, a distributed object storage system. ### `CDN_MINIO_LOGIN_ID` This variable represents the login ID for accessing Minio CDN. ### `CDN_MINIO_APP_ID` The `CDN_MINIO_APP_ID` environment variable is typically a unique identifier for your Minio CDN application. It will be provided by minio when you create an application. ### `CDN_MINIO_BUCKET_ID` This variable specifies the ID or name of the bucket within Minio CDN where your application stores objects. Buckets are used to organize and manage your data within Minio. ### `CDN_MINIO_FA_KEY` The `CDN_MINIO_FA_KEY` is a token or key used for authentication and authorization with Minio CDN. It may be provided by your CDN service provider. ### `CDN_MINIO_FA_URL` The `CDN_MINIO_FA_URL` is the URL used for authentication with Minio CDN. This URL is typically provided by your CDN service provider and is used to authenticate and authorize requests to your CDN resources. ### `CDN_MINIO_PASS` The `CDN_MINIO_PASS` environment variable represents the password or secret associated with your Minio CDN account. This password is used for secure access to your CDN resources. ### `CDN_MINIO_URL` The `CDN_MINIO_URL` environment variable specifies the base URL for accessing objects stored in Minio CDN. It represents the endpoint for your CDN service. ### 5. Fusionauth Environment Variables: #### FUSIONAUTH\_POSTGRES\_USER * Postgres user for Fusionauth service. #### FUSIONAUTH\_POSTGRES\_DBNAME * Postgres database name for Fusionauth service. #### FUSIONAUTH\_POSTGRES\_PASSWORD * Postgres password for Fusionauth service. #### FUSIONAUTH\_POSTGRES\_PORT * Postgres port for Fusionauth service. #### FUSIONAUTH\_DATABASE\_USERNAME * Database username for Fusionauth service. #### FUSIONAUTH\_DATABASE\_PASSWORD * Database password for Fusionauth service. #### FUSIONAUTH\_APP\_KICKSTART\_FILE * Kickstart file for Fusionauth application. #### FUSIONAUTH\_API\_KEY * API key for Fusionauth. #### ES\_JAVA\_OPTS * JVM options for Elasticsearch. #### FUSIONAUTH\_MEMORY * Memory allocation for Fusionauth. #### FUSIONAUTH\_APP\_MEMORY * Memory allocation for Fusionauth application. #### FUSIONAUTH\_ADMIN\_PASSWORD * Admin password for Fusionauth. #### FUSIONAUTH\_ADMIN\_EMAIL * Admin email for Fusionauth. ### 6. ODK Environment Variables: #### ODK\_POSTGRES\_PASSWORD * Postgres password for ODK. #### ODK\_POSTGRES\_USER * Postgres user for ODK. #### ODK\_POSTGRES\_DB * Postgres database for ODK. ### 7. Environment Variables for UCI-APIs: ### Postgresql #### PSQL\_DB\_URL * Postgres database URL for UCI APIs. #### UCI\_API\_POSTGRES\_PASSWORD * Postgres password for UCI APIs. #### UCI\_API\_POSTGRES\_USER * Postgres user for UCI APIs. #### UCI\_API\_POSTGRES\_DB * Postgres database for UCI APIs. #### POSTGRES\_HOST * Hostname for PostgreSQL. #### POSTGRES\_USER * PostgreSQL user. #### POSTGRES\_PASSWORD * PostgreSQL password. #### POSTGRES\_DB * PostgreSQL database. ### 8. Kafka #### KAFKA\_HOST\_DEV * Hostname for Kafka development environment. #### KAFKA\_USER * Kafka user. #### KAFKA\_PASS * Kafka password. #### KAFKA\_HOST * Hostname for Kafka. #### KAFKA\_PORT * Kafka port. #### KAFKA\_LOG\_LEVEL * Kafka log level. #### ODK\_SERVICE * URL for the ODK service. #### ODK\_USERNAME * ODK username. #### ODK\_PASSWORD * ODK password. #### ODK\_BASE\_URL * Base URL for ODK. #### FUSIONAUTH\_URL * URL for Fusionauth in UCI. #### FUSIONAUTH\_KEY * Fusionauth key. #### ADMIN\_TOKEN * Admin token. #### FUSIONAUTH\_ANONYMOUS\_BOT\_APP\_ID * Application ID for anonymous bot in Fusionauth. #### FUSIONAUTH\_ADMIN\_PASS * Admin password for Fusionauth. #### AUTH\_PUBLIC\_KEY * Public key for authentication. #### POSTHOG\_BASE\_URL * Base URL for PostHog. #### POSTHOG\_BATCH\_SIZE * Batch size for PostHog. #### POSTHOG\_FLUSH\_INTERVAL * Flush interval for PostHog. ### 9. Hasura #### HASURA\_GRAPHQL\_DATABASE\_URL * Database URL for Hasura GraphQL. #### HASURA\_GRAPHQL\_ENABLE\_CONSOLE * Enable Hasura GraphQL console. #### HASURA\_GRAPHQL\_ENABLED\_LOG\_TYPES * Enabled log types for Hasura GraphQL. #### HASURA\_GRAPHQL\_ADMIN\_SECRET * Admin secret for Hasura GraphQL. #### UCI\_CORE\_BASE\_URL * Base URL for UCI Core. #### TRANSFORMER\_BASE\_URL * Base URL for the transformer. #### GRAPHQL\_BASE\_URL * Base URL for GraphQL in UCI API DB. #### INBOUND\_URL * URL for inbound service. ### 10. Encryption Key #### ENCRYPTION\_KEY * Encryption key. #### ENV * Environment (e.g., prod). ### 11. UCI Core #### CAMPAIGN\_URL * URL for campaigns in UCI Core. #### CAMPAIGN\_ADMIN\_TOKEN * Admin token for campaigns in UCI Core. ### 12. Other Kafka Configurations #### BOOTSTRAP\_SERVERS * Bootstrap servers for Kafka. #### KAFKA\_INBOUND\_PROCESSED\_TOPIC * Kafka topic for inbound processed messages. #### KAFKA\_CAMPAIGN\_TOPIC * Kafka topic for campaigns. #### KAFKA\_INBOUND\_UNPROCESSED\_TOPIC * Kafka topic for inbound unprocessed messages. #### KAFKA\_INBOUND\_GS\_OPTED\_OUT\_TOPIC * Kafka topic for inbound GS opted-out messages. #### KAFKA\_INBOUND\_ERROR\_TOPIC * Kafka topic for inbound error messages. #### KAFKA\_OUTBOUND\_TOPIC * Kafka topic for outbound messages. #### KAFKA\_TELEMETRY\_TOPIC * Kafka topic for telemetry. #### KAFKA\_ODK\_TRANSFORMER\_TOPIC * Kafka topic for ODK transformer. #### KAFKA\_ODK\_TRANSFORMER\_TOPIC\_PATTERN * Kafka topic pattern for ODK transformer. #### KAFKA\_LOGS\_TOPIC * Kafka topic for telemetry logs. #### KAFKA\_BROADCAST\_TRANSFORMER\_TOPIC * Kafka topic for broadcast transformer. #### KAFKA\_PROCESS\_OUTBOUND * Kafka topic for processing outbound messages. #### KAFKA\_MESSAGE\_REPORT\_TOPIC * Kafka topic for message reports. #### KAFKA\_GENERIC\_TRANSFORMER\_TOPIC * Kafka topic for generic transformer. #### KAFKA\_NOTIFICATION\_TOPIC * Kafka topic for notification outbound. #### KAFKA\_NOTIFICATION\_INBOUND\_PROCESSED * Kafka topic for processed inbound notifications. ### 13. FormsDB Configurations #### FORMS\_DB\_HOST * Hostname for FormsDB. #### FORMS\_DB\_NAME * Database name for FormsDB. #### FORMS\_DB\_PASSWORD * Database password for FormsDB. #### FORMS\_DB\_PORT * Database port for FormsDB. #### FORMS\_DB\_URL * Database URL for FormsDB. #### FORMS\_DB\_USERNAME * Database username for FormsDB. ### 14. ODK Configurations #### ODK\_PASS * ODK password. #### ODK\_URL * URL for ODK. #### ODK\_USER * ODK user. ### 15. Cassandra COnfigurations #### CASSANDRA\_URL * Cassandra URL. #### CASSANDRA\_PORT * Cassandra port. #### CASSANDRA\_KEYSPACE * Cassandra keyspace. #### CASSANDRA\_MIGRATION\_COUNT * Cassandra migration count. #### CASSANDRA\_PASSWORD * Cassandra password. ### 16. Ports #### INBOUND\_INTERNAL\_PORT * Internal port for inbound service. #### INBOUND\_EXTERNAL\_PORT * External port for inbound service. #### OUTBOUND\_INTERNAL\_PORT * Internal port for outbound service. #### ORCHESTRATOR\_INTERNAL\_PORT * Internal port for orchestrator service. #### TRANSFORMER\_INTERNAL\_PORT * Internal port for transformer service. #### BROADCAST\_TRANSFORMER\_INTERNAL\_PORT * Internal port for broadcast transformer service. ### 17. Assesment Check Characters #### ASSESSMENT\_GO\_TO\_START\_CHAR * Character for going to the start of an assessment. #### ASSESSMENT\_ONE\_LEVEL\_UP\_CHAR * Character for moving one level up in an assessment. #### CURRENT\_VERSION * Current version. #### XMSG\_USER\_DATA\_BEFORE\_HOUR * User data before a certain hour. ### 18. PWA Adapter #### PWA\_TRANSPORT\_SOCKET\_BASE\_URL * Base URL for PWA transport socket. #### TRANSPORT\_SOCKET\_BASE\_URL * Base URL for transport socket. ### 19. Netcore #### NETCORE\_WHATSAPP\_AUTH\_TOKEN * Authorization token for Netcore WhatsApp. #### NETCORE\_WHATSAPP\_SOURCE * Netcore WhatsApp source. #### NETCORE\_WHATSAPP\_URI * Netcore WhatsApp URI. ### 20. Redis #### REDIS\_DB\_INDEX * Redis database index. #### REDIS\_HOST * Hostname for Redis. #### REDIS\_PORT * Redis port. #### REDIS\_NUMBER\_PORT * Redis number port. #### REDIS\_PASS * Redis password. #### REDIS\_ENABLED * Enable or disable Redis. #### TRANSPORT\_SOCKET\_CACHE\_PORT * Port for transport socket cache. #### TRANSPORT\_SOCKET\_CACHE\_HOST * Hostname for transport socket cache. ### 21. CDN Config #### SELECTED\_FILE\_CDN * Selected file CDN. #### TEMPLATE\_SERVICE\_BASE\_URL * Base URL for the template service. #### CDAC\_BASE\_URL * Base URL for CDAC. ### 22. Vault Credentials #### VAULT\_FUSION\_AUTH\_URL * URL for Vault Fusionauth. #### VAULT\_FUSION\_AUTH\_TOKEN * Token for Vault Fusionauth. #### VAULT\_SERVICE\_URL * URL for Vault service. #### VAULT\_SERVICE\_TOKEN * Token for Vault service. ### 23. Telemetry Events #### EXHAUST\_TELEMETRY\_ENABLED * Enable or disable exhaust telemetry. #### POSTHOG\_EVENT\_ENABLED * Enable or disable PostHog events. #### TELEMETRY\_EVENT\_PDATA\_ID * Telemetry event PData ID. #### POSTHOG\_TELEMETRY\_APIKEY * API key for PostHog telemetry. #### POSTHOG\_TELEMETRY\_URL * URL for PostHog telemetry. ### 24. Doubtnut config #### DOUBTNUT\_BASE\_URL * Base URL for DoubtNut. #### DOUBTNUT\_WELCOME\_MSG * Welcome message for DoubtNut. #### DOUBTNUT\_WELCOME\_VIDEO * Welcome video URL for DoubtNut. #### DOUBTNUT\_AUTH\_KEY * Authentication key for DoubtNut. ### 25. FormsDB Graphql Admin Secret #### FORMSDB\_HASURA\_ADMIN\_SECRET * Admin secret for FormsDB Hasura GraphQL. ### 26. AKHQ #### AKHQ\_USERNAME * Username for AKHQ. #### AKHQ\_PASSWORD * Password for AKHQ. ### 27. VAULT for UCI #### VAULT\_TOKEN * Token for Vault. #### NOTIFICATION\_KEY\_ENABLE * Enable or disable notification key. ### 28. Email Config #### EMAIL\_HOST * Hostname for email. #### EMAIL\_USERNAME * Email username. #### EMAIL\_PASSWORD * Email password. #### EMAIL\_PORT * Email port. #### RECIPIENT\_EMAILS * Recipient emails. ### 29. BROADCAST\_NOTIFICATION\_CHUNK\_SIZE * Chunk size for broadcast notifications. ### 30. Caffeine cache values #### CAFFEINE\_CACHE\_MAX\_SIZE * Maximum size for Caffeine cache. #### CAFFEINE\_CACHE\_EXPIRE\_DURATION * Expiry duration for Caffeine cache. ### 31. MINIO #### MINIO\_MEDIA\_UPLOAD\_URL * URL for Minio media upload. #### MINIO\_GET\_SIGNED\_FILE\_URL * URL for Minio get signed file. ### 32. Redis Name for Notification Orchestrator #### NOTIFICATION\_KAFKA\_CACHE * Kafka cache name for notifications. ### 33. JVM Size #### JVM\_ARGS\_INBOUND * JVM arguments for inbound service. #### JVM\_ARGS\_ORCHESTRATOR * JVM arguments for orchestrator service. #### JVM\_ARGS\_TRANSFORMER * JVM arguments for transformer service. #### JVM\_ARGS\_BROADCAST\_TRANSFORMER * JVM arguments for broadcast transformer service. #### JVM\_ARGS\_OUTBOUND * JVM arguments for outbound service. ### 34. BotService Utils #### WEBCLIENT\_INTERVAL * Interval for web client. #### WEBCLIENT\_RETRY\_MAX\_ATTEMPTS * Maximum retry attempts for web client. #### WEBCLIENT\_RETRY\_MIN\_BACK\_OFF * Minimum back-off time for web client retries. ### 35. Redis Data Timeout(Second) #### REDIS\_KEY\_TIMEOUT * Redis key timeout in seconds. ### 36. Outbound Notification and Cassandra Reactive Buffer Size and Time Config #### OUTBOUND\_NOTIFICATION\_BUFFER\_MAX\_SIZE * Maximum size for outbound notification buffer. #### OUTBOUND\_NOTIFICATION\_BUFFER\_MAX\_TIME * Maximum time for outbound notification buffer. #### OUTBOUND\_CASSANDRA\_BUFFER\_MAX\_SIZE * Maximum size for outbound Cassandra buffer. #### OUTBOUND\_CASSANDRA\_BUFFER\_MAX\_TIME * Maximum time for outbound Cassandra buffer. ### 37. Transformer ODK Postgres Pool Config #### FORMS\_DB\_POOL\_INITIAL\_SIZE * Initial pool size for FormsDB. #### FORMS\_DB\_POOL\_MAX\_SIZE * Maximum pool size for FormsDB. ### 38. TRANSACTION\_AUTH\_TOKEN #### AUTHORIZATION\_KEY\_TRANSACTION\_LAYER * Authorization key for the transaction layer. ### 39. Logging #### LOGS\_FOLDER * Folder for logs. #### ANALYSIS\_TRIGGGER\_FILE * Analysis trigger file. ### 40. INBOUND CAFFINE #### CAFFINE\_INVALIDATE\_ENDPOINT * Endpoint to invalidate Caffeine cache. ### 41. Broadcast Report #### BROADCAST\_BOT\_REPORT\_ENDPOINT * Endpoint for bot report in broadcast. ### 42. Web-Channel #### REACT\_APP\_TRANSPORT\_SOCKET\_URL * URL for React app transport socket. #### REACT\_APP\_UCI\_BOT\_BASE\_URL * Base URL for the UCI bot in React app. #### REACT\_APP\_CHAT\_HISTORY\_URL * URL for chat history in React app. #### REACT\_APP\_OTP\_BASE\_URL * Base URL for OTP in React app. #### REACT\_APP\_OWNER\_ID * Owner ID in React app. #### REACT\_APP\_OwnerOrgId * Owner organization ID in React app. #### REACT\_APP\_AUTH\_TOKEN * Authentication token in React app. #### REACT\_APP\_MOBILE * Mobile number in React app. #### REACT\_APP\_Admin\_Token * Admin token in React app. #### REACT\_APP\_FILTER\_LIST * Filter list flag in React app. #### FORMSDB\_HASURA\_GRAPHQL\_ADMIN\_SECRET * Admin secret for FormsDB Hasura GraphQL. #### FUSIONAUTH\_CLIENT\_SECRET * Client secret for Fusionauth. #### FUSIONAUTH\_APPLICATION\_ID * Application ID for Fusionauth. #### FUSIONAUTH\_DATABASE\_USER * Database user for Fusionauth. ### 43. For Authentication from NL #### TRANSPORT\_SOCKET\_JWT\_AUTH\_URL * JWT authentication URL for transport socket. # Developer - Contribute to UCI I am an open-source enthusiast and I want to contribute to the Sunbird UCI [**UCI Basics**](/use/developer/uci-basics) As a new developer, before you contribute to UCI, you should get familiar with a few internal basics about UCI This section dives deep into the basics of UCI, setting you up for a good, contribution journey. [**Development Environment**](/use/developer/development-environment) This section helps any developer to get ready with the local setup of the codebase to start their contribution journey. [**API Documentation**](/use/developer/api-documentation) This section covers relevant APIs required for different use cases for any adopter in their journey of using UCI. [**Database Schema**](/use/developer/database-schema) This section provides essential information regarding the Database Schema necessary for various use cases, assisting any adopter throughout their UCI utilization journey. [**Contribution Guide**](/use/developer/contribution-guide) This section covers a long range of items, from the basics of how you contribute your first PR to how you contribute a new adapter and a new transformer. *** *** # UCI Basics ## 1. Overview UCI is a multi-channel communication management engine, designed to be extremely modular and configurable. UCI has a total of 9 repositories which are being used to process the bot messages from inbound to outbound. We have 5 main repositories who will process the messages from receiving the messages from channel(Sms/Whatsapp), converting the message to xmessage, authenticating the user, producing the next message, converting it back to message format for channel and sending it to the channel(Sms/Whatsapp). The other repositories are the helper repositories with each doing different jobs. ## 2. Brief Description A brief description about the repositories included are listed below. ### **2.1. Utils** Utils communicate with the federation service via apis. It holds all the utility/helper classes which will be used in other repositories. ### **2.2. Message Rosa** Message Rosa holds all the core models for XMessage & its property fields. ### **2.3. Dao** Dao holds the dao & repository classes for XMessage. ### **2.4. Adapter** Adapters convert information provided by channels (SMS, Whatsapp) for each specific provider to xMessages and vice versa. Adapters are gateway to the external services and are responsible for receiving user response and sending the response to users. [Click here](/use/developer/uci-basics/adapters) to see the detailed documentation of adapter. ### **2.5. Inbound** Inbound receives the messages from a channel, and uses the channel adapter to convert it to XMessage format. Once the message is converted, it will be pushed to the kafka topic, the orchestrator will listen to this topic to further process it. ### **2.6. Orchestrator** Orchestrator authenticates & processes the user data from whom the message is received. The XMessage will then be pushed to the transformers topic based on the conversation configuration, the corresponding transformer(transformer/broadcast-transformer) will listen to this topic to further process it. ### **2.7. Transformer** This is a ODK transformer, which transforms the previous xMessage from the user to one that needs to be sent next. It is a microservice that returns a new xMessage based on the previous user action, which will then be shown to the user. The XMessage will be pushed to the process-outbound topic, orchestrator listens to this topic and will further process it to & send it to outbound topic. [Click here](/use/developer/uci-basics/transformers) to see the detailed documentation of adapter. ### **2.8. Broadcast Transformer** Broadcast transformer transforms the single message to multiple messages for each user segment associated with a conversation. It is a micro service that generated multiple XMessage and push it to process-outbound topic, orchestrator listens to this topic and will further process it to & send it to outbound topic. ### **2.9. Outbound** Outbound converts the xMessage to the one that will be sent to the channel(sms/whatsapp). It will then be sent to the network provider(Netcore/Gupshup) who will send it to the channel. ## 3. Basics To have a more deeper understanding of UCI, we have mentioned a few features included & specification here. * [XMessage Specification](/use/developer/uci-basics/xmessage-specification) * [Transformers](/use/developer/uci-basics/transformers) * [Adapters](/use/developer/uci-basics/adapters) * [User Segment](/use/developer/uci-basics/user-segment) * [Schema Overview](/use/developer/uci-basics/schema-overview) # XMessage Specification ## XMessage Specification ### 1. Introduction The XMessage specification is used by tools in the [Communication package](https://samagra-development.github.io/docs/docs/COConversations) ecosystem. It is a subset of the far larger [XMPP](https://xmpp.org/) and also contains a few additional features not found in the XMPP specification. The purpose of this specification is to provide a common message description standard that many different kinds of compatible tools can be based on. Using a single, shared message description standard has the following advantages: 1. Users in the Communication package can mix and match tools and reassess which they use based on their changing needs. In particular, they don't get locked in to tools that may become deprecated or for which an attractive replacement becomes available. 2. Tool implementors in the Communication ecosystem can benefit from feedback from a broad range of collaborators when designing new core functionality. 3. Tool implementors in the Communication ecosystem can share core implementations. 4. Enables building a message transport protocol between services. This document is intended primarily for developers who build message processing engines (Transformers) or rule engines (Orchestrator). The document assumes at least a fair understanding of XML. It is also useful to refer to [XMPP](https://xmpp.org/) for details about shared features. The same features can be built over json as well. #### 1.1 XML vs JSON Even though JSON is tempting we chose XML for data interchange for the following reasons, 1. Its is a document markup language which is what most of the text is for various modes of communication 2. It has a schema and allows for validation of the same 3. Building contracts among servers are a breeze 4. It has namespaces which are used when combining different format. (Although we are not using namespaces right now, this spec will evolve into using namespaces for various sections as it expands) ### 2. Structure The current structure of XMessages is divided into following sections. * Sender/Reciever Information * Channel/Provider Information * Transformers that need to act on this message * Context related parameters * Message body Let's dive deeper into all of these, but before that, this is the complete xml schema (this is used for validation currently) ```xml ``` #### 2.1 Sender/Reciever Information The `from` and `to` construct uses the structure show below. ```xml ``` | Parameter | Description | | ------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `bot` | Refers to if the response/reply from/to the user is from a bot or an actual user | | `broadcast` | Is for use case where the user or sever is sending the same message to multiple users or a single one | | `userID` | Current User ID; When the message is still not parsed, it could be his phone number but should generally be a UUID which represents the user in the database | | `campaignID` | ID of the conversation campaign the message is part of | | `formID` | Ff the reply is for a ODK Form based bot, this represents the ID of that form | #### 2.2 Channel/Provider Information ```xml ``` | Parameter | Description | | ------------- | ---------------------------------------------------------- | | `contentURI` | Channel for the message (SMS, Email, WhatsApp etc) | | `providerURI` | Provider for the message (Gupshup, AmazonSES, Twilio, etc) | #### 2.3 Transformers that need to act on this message Each message can be acted on by multiple transformers to modify/reply the payload/channel etc. This happens through `Transformers`. Each transformer is assigned an ID when it is registered. ```xml ``` | Parameter | Description | | --------- | --------------------------------------------------------- | | `id` | Transformer's ID (genereated at the time of registration) | | `meta` | Meta data for the transformer | #### 2.4 Context related parameters This section describes the ```xml ``` | Parameter | Description | | ------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `app` | App is a global namespace the message is part of. For example an app may correspond to multichannel conversation that many users are part of. Any segreation of conversation is defined as an app | | `userstate` | Any of the following => `typing`, `offline`, `unregistered`, `registered`, `active`, `inactive` | | `messageState` | State of a message sent to the user. Any of the following => `FAILED_TO_DELIVER`, `DELIVERED`, `QUEUED`, `READ`, `REPLIED` | | `conversationStage` | Stage of the current conversation. Any of the following => `STARTING`, `ONGOING`, `COMPLETED` | | `lastMessageID` | ID of the last message sent for this conversation thread. | | `thread` | The thread which the message belongs to | #### 2.5 Message body Message body referes to the actual message body. The current list of message types includes plain text, xHTML, location, media, or a contactCard. ```xml ``` ### 3. Styling There are two supported protocols for message styling - [XEP 0393](https://xmpp.org/extensions/xep-0393.html) and [XEP 0071](https://xmpp.org/extensions/xep-0071.html). Note only message inside the `` are to be implemented with the following rules. All other tags are escaped. The common use cases and examples of messages are as follows: #### 3.1 Bold ```xml (Two spans, both )(*alike in dignity*) ``` Should be styled as => (Two spans, both )(**alike in dignity**) #### 3.2 Strikethrough ```xml Everyone ~dis~likes cake ``` Should be styled as => Everyone ~~dis~~likes cake #### 3.3 Italics ```xml The full title is _Twelfth Night, or What You Will_ but _most_ people shorten it. ``` Should be styled as => The full title is *Twelfth Night, or What You Will* but *most* people shorten it. #### 3.4 Monospace Text enclosed by a '\`' (U+0060 GRAVE ACCENT) is a preformatted span SHOULD be displayed inline in a monospace font. A preformatted span may only contain a single plain span. ```xml Wow, I can write in `monospace`! ``` Should be styled as => Wow, I can write in `monospace`! #### 3.5 Plain Any text inside of a block that is not part of another span is implicitly considered to be inside of a "plain text" span. ```xml (There are three blocks in this body marked by parens,) (but there is no *formatting) (as spans* may not escape blocks.) ``` #### 3.6 Escaping On rare occasions styling hints may conflict with the contents of a message. For example, if the user sends the emoji `"> _ <"` it would be interpreted as a block quote. Senders may indicate to the receiver that a particular message SHOULD NOT be styled by adding an empty `` element. ```xml > _ < ``` #### 3.7 Implementation Details This document does not define a regular grammar and thus styling cannot be matched by a regular expression. Instead, a simple parser can be constructed by first parsing all text into blocks and then recursively parsing the child-blocks inside block quotations, the spans inside individual lines, and by returning the text inside preformatted blocks without modification. It is RECOMMENDED that styling directives be displayed and formatted in the same manner as the text they apply to. For example, the string "*emphasis*" would be rendered as "*emphasis*". This specification does not provide a mechanism for removing styling from individual spans or blocks within a styled message. Implementations are free to implement their own workarounds, for example by inserting Unicode non-printable characters to invalidate styling directives, but no specific technique is known to be widely supported. ### 4. FAQs To be updated based on incoming feedback. Feel free to write into in case you have questions, feedback or want to know more! # Transformers ## 1. Overview A transformer is a stateless processing object that takes inputs and converts the input into a processed response. Transformers may in turn call external services if needed. We currently have two working transformers which are listed below. * [ODK Transformer](/use/developer/uci-basics/transformers/odk-transformer) * [Broadcast Transformer](/use/developer/uci-basics/transformers/broadcast-transformer) ## 2. How to contribute a transformer [Click here](/use/developer/contribution-guide/create-a-transformer) to see the detailed documentation on how to create your own transformer. # ODK Transformer ### 1. Overview This transformer uses odk forms to get the input from user and output the next reply to be sent. ODK forms contains the questions and flow of the questions. ### 2. Setup ODK Aggregate [ODK Aggregate](https://docs.getodk.org/aggregate-intro/) is an open source Java application that stores, analyzes, and presents XForm survey data collected using ODK Collect. It supports a wide range of data types, and is designed to work well in any hosting environment. With Aggregate, your team can: * Host blank XForms used by ODK Collect or other OpenRosa clients * Store and manage XForm submission data * Visualize collected data using maps and simple graphs * Export and publish data in a variety of formats Aggregate can be hosted on cloud providers such as DigitalOcean, and Amazon Web Services, or your own local or cloud server. There's also a pre-configured virtual machine image that is ready to deploy on any computer. Please check this link to find [how to set up ODK Aggregate?](https://docs.getodk.org/aggregate-setup/) Refer this link to find steps to [use ODK.](https://docs.getodk.org/aggregate-use/) ### 3. How it works: It saves all the question & assessment data in db and fetch the same when needed. This transformer uses MenuManager class to handle all ODK related processes. MenuManager uses [javarosa](https://mvnrepository.com/artifact/org.getodk/javarosa/3.3.0) library. It loads the odk form, fetches the next question based on previous xpath or no path, validate the answer to current question & generate the error messages if any. A form is first loaded to the `FECWrapper` class. It gives us the `FormEntryController` class object to handle ODK related functionlity. Some of the main methods we use to fetch the relavent data using this class are given below. 1. Get current form index ```java FormController.getModel().getFormIndex() ``` 2. Set the current index to previous event of the form ```java FormController.stepToPreviousEvent() ``` 3. Set the current index to next event of the form ```java FormController.stepToNextEvent() ``` 4. TreeReference of the fully qualified element of the form ```java FormController.getModel().getFormIndex().getReference() ``` 5. Get the text for question at current index ```java FormController.getModel().getQuestionPrompt().getQuestionText() ``` 6. Get the type for question (Eg. select/input/audio/image etc.) ```java FormController.getModel().getQuestionPrompt().getControlType() ``` FormEntryController has different event types based on the ODK form, from which some are listed below. 1. Represents a question: `EVENT_QUESTION` 2. Represents a question group: `EVENT_GROUP` 3. Represents a repeat question: `EVENT_REPEAT` ### 4. MenuManager Methods **1. start** This method loads the odk form, process the current answer, identify the next question/reponse and returns a `ServiceResponse` which includes a few properties that will be used by the ODK Transformer. 1. Current xpath 2. XMessage payload for Next question with question text 3. Updated Instance XML 4. Form Version 5. Form ID 6. Next Question object **2. addResponseToForm** It validates the current answer against the type of question (input/select choice/media etc.) & returns a status for the same. **3. getIndexFromXPath** It gets the index from the give xpath. **4. getXPath** It gets the xpath from the current form index. **5. renderQuestion** Renders a question by cleaning its text & adding help text to it, if there is any. **6. createView** Creates and returns a new view based on the event type passed in. It returns the XMessage payload with question text and its choices if the question is of select choice type. **7. loadForm** It loads a ODK xml form from the given path. It creates form defination & initialize the `FormDef`, `FormEntryModel`, `FormEntryController` class objects to handle all form related functionalities. ### 5. ODK Consumer Reactive(ODK Transformer) Methods **1. onMessage** It listens to the transformer kafka topic. Once a messages has been published to the kafka topic, it consumes the message, transform the message to get the next reply for user & send it to the next kafka topic. **2. transform** It takes in the xMessage and returns a next XMessage to be send to user. It uses the MenuManager class to fetch the relavant data like xpath, next question etc. & validate the answer from user. **3. updateQuestionAndAssessment** It fetches the previous question from database. If it does not exists in the database, it will first save the question then save the assessment data against the question. **4. saveAssessmentData** It saves the assessment data in the database and generate a telemetry event to be send to the telemetry topic. Any service can then consume this telemetry event to process & generate reports for the assessment. **5. decodeXMessage** It saves the gupshup message entity & gupshup state entity with properties user phone, message, xpath, xml etc. It will return a XMessage with next message properties. ### 6. New features We have introduced a few new features which are listed below. #### 6.1 Interactive List & Buttons We have added this feature to enable a bot show its choice options as a list or buttons on whatsapp. We use styling tags for this, which are listed below 1. list 2. buttonsForListItems [Click here](/use/developer/uci-basics/transformers/odk-transformer/interactive-messages) to see how to add these in an ODK form. #### 6.2 Send Media to User We have added this feature to enable a bot to send media files to a user for platforms which allows media files. We use styling tags for this, which are listed below 1. image 2. audio 3. video 4. file [Click here](https://uci.sunbird.org/use/developer/uci-basics/transformers/pages/dJuiIms0XUrJ0hPnH3sD#1.-send-media-to-user) to see how to add these in an ODK form. #### 6.3 Receive Media from User We have added this feature to enable a bot to receive media files upload by a user. We use different type in ODK from to enable this. 1. image 2. audio 3. video 4. file [Click here](https://uci.sunbird.org/use/developer/uci-basics/transformers/pages/dJuiIms0XUrJ0hPnH3sD#2.-receive-media-from-user) to see how to add these in ODK form. #### 6.4 Receive Location from User We have added this feature to enable a bot to receive location of a user. For this we use a question type as geopoint. [Click here](/use/developer/uci-basics/transformers/odk-transformer/location) to see how to add this in ODK form. # Interactive-Messages ## Interactive Messaging The Interactive Message Templates feature allows you to add buttons in message templates that can be used with List and Quick Replies. Buttons will give the ability to develop interactive experiences with pre-set options for users.\ There are two types of buttons: ### 1. List Message List messages provide a simpler and more consistent format than text-based lists for people to find and select\ List messages are a way to allow users to easily choose from up to 10 options. #### 1.1 Steps for creating List Message For sending a List Message, add following properties in ODK form : * **type** : `select_one `\ Column *type* describes the property of any particular message. * **name** : ``\ Column *name* refers to unique *field\_name* of any message. * **label** : ``\ In label, we can write text content that we want to send to user, If there are any choices available for say *select one field*, the choices will be appended with this text content. * **bind::stylingTags** : `list`\ Bind tag indicates, how a *select choice* option should be displayed to user as list or buttons. It is also used to define the media type a message. * For creating options as list items we have to add the items in the *choices* tab as we do for the *select choices*. These are the properties that needs to be set: * **list name** : ``\ In list name, we write the field name, same as we wrote in the *name* field of survey tab. * **name** : ``\ In name field, we write index of a list item. * **label** : ``\ In label, we can write the title of a list item, which will be shown to the user. #### 1.2 Rules for creating List Message * In a single List Message maximum 10 list items are allowed. * Maximum 24 characters are allowed in list item label. * Alphanumeric characters and Spaces are allowed in list item label. #### 1.3 Example for ODK form of List Message * Survey Tab ![](https://1242145269-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FnC2042spZ2TIbqdRA37S%2Fuploads%2FyLHDqEE34Zklj4aLw1gI%2FListMessage_SurveyTab.png?alt=media\&token=5f44be83-8950-4cbf-896a-ace9df6f4e24) * Choices Tab ![](https://1242145269-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FnC2042spZ2TIbqdRA37S%2Fuploads%2F1Uf4YmphPJxphWYEVOka%2FListMessage_CoicesTab.png?alt=media\&token=79d4b034-29de-4a03-a2e0-2f19477dfc26) #### 1.4 Example of List Message Preview ![](https://1242145269-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FnC2042spZ2TIbqdRA37S%2Fuploads%2FPEKYgWlWjKYBZ1nT8gO5%2FListMessage_example.jpeg?alt=media\&token=0e005158-494c-4ba1-9abb-1bdabe9dafca) ### 2. Quick Reply Button These quick reply buttons will help improve the quality of conversations with users by prompting responses that can reduce spelling errors and improve an automated experience.\ These buttons can be attached to text messages and a maximum of 3 Buttons are allowed in one Message. #### 2.1 Steps for creating Quick Reply Button To send Quick Reply Button follow the below steps while creating ODK form : * **type** : `select_one <field_name>`\ Column *type* describes the property of any particular message. * **name** : `<field_name>`\ Column *name* refers to unique *field\_name* of any message. * **label** : `<text>`\ In label, we can write text content that we want to send to user, If there are any choices available for say *select one field*, the choices will be appended with this text content. * **bind::stylingTags** : `buttonsForListItems`\ Bind tag indicates, how a select choice option should be displayed to user as list or buttons. It is also used to define the media type a message. * For creating buttons for quick reply, we have to add the items in the choices tab as we do for the select choices. These are the properties that needs to be set: * **list name** : `<field_name>`\ In list name, we write the field name, same as we wrote in the *name* field of survey tab. * **name** : `<indexing>`\ In name field, we write index of a list item. * **label** : `<title>`\ In label, we can write the title of a button, which will be shown to the user. #### 2.2 Rules for creating Quick Reply Buttons * In a single Quick Reply Message, maximum 3 buttons are allowed. * Maximum 20 characters are allowed in button label. * Alphanumeric characters and Spaces are allowed in button label. #### 2.3 Example for ODK form of Quick Reply Button * Survey Tab ![](https://1242145269-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FnC2042spZ2TIbqdRA37S%2Fuploads%2FmeX1LiRrS6iNBcl1v844%2FQuickReplyButton_SurveyTab.png?alt=media\&token=da58cfc1-9c8e-4700-9913-31087935cdda) * Choices Tab ![](https://1242145269-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FnC2042spZ2TIbqdRA37S%2Fuploads%2FBRA48QttR5BXMpnAdQRQ%2FQuickReplyButton_ChoicesTab.png?alt=media\&token=227fdf9f-e26b-4498-a846-ac341bf3cdb8) #### 2.4 Example of Quick Reply Button Preview ![](https://1242145269-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FnC2042spZ2TIbqdRA37S%2Fuploads%2FAgUh98Lf7mivrJ8apTNj%2FQuickReplyButton_Example.jpeg?alt=media\&token=0867b1d9-a605-46ec-ac62-5a234d86c787) # Media Messages ## Overview UCI provides an option to send media files to user or receive the media files from user. We use different types of styling tags to send media files to user. ### 1. Send media to user We use below styling tags to send media files to user. 1. image 2. audio 3. video 4. file (Document files only) As the name says these tags are used to specify the media type to be sent to user. These styling tags will be added **Steps for adding media message** For sending a media to user, add following properties in ODK form : * **type** : `select_one <field_name>`\ Column *type* describes the property of any particular message. * **name** : `<field_name>`\ Column *name* refers to unique *field\_name* of any message. * **label** : `<text>`\ In label, we can write text content that we want to send to user. For media files mention the file name that you are uploading inside {} placeholders. For example, if your file name is 'test.png', set the label as {test.png}. * **bind::stylingTags** : `image/audio/video/file`\ This column indicates, which type of media file should be displayed to user. * **bind::caption** : \<caption>\ This column has the caption for media file specifically for image/file. As for whatsapp the caption is required. **ODK Styling tag example:** ![](https://1242145269-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FnC2042spZ2TIbqdRA37S%2Fuploads%2FWt01PJRe9vhEmhm1Vuk8%2Fmedia-styling-tag.png?alt=media\&token=aaf99e68-d17b-4a37-a7e2-2ae3268726e5) **Whatsapp Preview** ![](https://1242145269-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FnC2042spZ2TIbqdRA37S%2Fuploads%2FX1dLDiHAzpq5fledzLH6%2Fmedia-image-whatsapp-preview.png?alt=media\&token=24ee36f5-df48-4109-8428-a4b8d37883f3) ### 2. Receive media from user In ODK forms we can specify which type of content we want from the user. For media files, we should mention the type for media ones. Below are the types for media files. 1. image 2. audio 3. video 4. file **2.1 Steps for expecting media message** For receiving media from user, add following properties in ODK form : * **type** : `image/audio/video/file`\ Column *type* describes the property of any particular message. * **name** : `<field_name>`\ Column *name* refers to unique *field\_name* of any message. * **label** : `<text>`\ In label, we can write text content that we want to send to user. **2.2 Allowed files types** | Field Type | Allowed Media Type | | ---------- | ------------------------------------------------------------------------------------------------------------ | | Image | image/jpg, image/jpeg, image/gif, image/png | | Audio | audio/mp3, audio/aac, audio/wav, audio/flac, audio/ogg, audio/ogg; audio/wma, audio/x-ms-wma, audio/mpeg | | Video | video/mp4, video/flc, video/mov, video/wmv, video/mkv, video/quicktime, video/x-matroska, video/x-flv | | File | application/pdf, application/msword, application/vnd.openxmlformats-officedocument.wordprocessingml.document | **2.3 ODK media type example** ![](https://1242145269-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FnC2042spZ2TIbqdRA37S%2Fuploads%2FzHE96gVBTUQQYvlRjZtf%2Fupload-media-odk.png?alt=media\&token=e4809a33-4070-4e6a-a5a3-b486f51b7748) # Location ## Overview UCI provides an option to receive location from a user. We use question type as geopoint for this. ## Receive **location** from user **Steps for expecting location** For receiving media from user, add following properties in ODK form : * **type** : geopoint\ Column *type* describes the property of any particular message. * **name** : `<field_name>`\ Column *name* refers to unique *field\_name* of any message. * **label** : `<text>`\ In label, we can write text content that we want to send to user. **ODK location type example** ![](https://1242145269-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FnC2042spZ2TIbqdRA37S%2Fuploads%2F7NE7Yf1wXAONW62TVLoa%2Freceive-location-odk.png?alt=media\&token=ea56615f-1a18-4ea2-8633-23cdf4efe9f7) **Whatsapp Preview** ![](https://1242145269-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FnC2042spZ2TIbqdRA37S%2Fuploads%2FNtZl92zCD28bEeYsxE93%2Freceive-location-whatsapp-preview.png?alt=media\&token=ee5f6187-96e3-49fb-9030-b360c7ffa7b5) # Broadcast Transformer ## Overview Broadcast transformer converts a single message to multiple messages. Orchestrator send a list of user with there messages to be sent in a single XMessage. This transformer will generate mutiple xMessages based on the list of users & messages. It then pushes these messages to process outbound topic. Orchestrator gets to decide if any other change is required for these messages or to send it to directly to user via outbound. ## Configuration A bot will work with a broadcast transformer when we mention the type of the transformer as **broadcast.** When a bot is added, certain configuration will be added with this bot. This configuration needs to be changed for broadcast transformer. A sample broadcast configuration is given below ``` { "id": "774cd134-6657-4688-85f6-6338e2323dde", "meta": { "body": "Hi Sanchita! ${name} has submitted an air travel request for ${phoneNo}.", "type": "JS_TEMPLATE_LITERALS", "params": ["name", "phoneNo"] }, "type": "broadcast" } ``` Below are the brief details of each parameters included. 1. **id**: id of broadcast transformer 2. **meta:** Data related to message that will be to users. 1. **body**: Message template text with parameters in it. Parameters should be included in the template in as **"${name}"**. 2. **type**: Type of message template. **Eg**. JS\_TEMPLATE\_LITERALS 3. **params**: Array of parameters provided in user segment. These parameter's values from user segment will be replaced in the message template text. 3. **type**: Type of transformer. **Eg**. broadcast This configuration will be used when create a conversation logic. [Click here](https://uci.sunbird.org/use/developer/uci-basics/transformers/pages/iyh1Cgn2i7ue92G4ym2v#2.2-create-a-conversation-logic) to see the conversation logic api. ## User segment User segment needs to be defined to broadcast a message to users. This user segment must have a **phoneNo** parameter in it. The user segment can have multiple parameters in it, these parameters will be used later to be replaced in the message template text before sending the message to users. Below is a sample of user segment. ``` { "data": [ { "id": "1", "phoneNo": "9877777777", "name": "John", "url": "http://google.com" }, { "id": "2", "phoneNo": "9866666666", "name": "Mike", "url": "http://google.com" } ] } ``` # Adapters ## 1. Conversation Channel Adapters Adapters convert information provided by channels (SMS, Whatsapp) for each specific provider to xMessages and vise versa. Adapters are gateway to the external services and resposible to recieving user response and sending response to users. Thus the two major functions of Adapters are * Convert API/webhook data from channel (and provider) to xMessages * Convert xMessages back to API/webhook data format for the specific channel(and provider) A simplified diagram of what adapters do is shown below. ![](https://1242145269-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FnC2042spZ2TIbqdRA37S%2Fuploads%2FkIo3hjgtdIgMj0Zw8Dgo%2Fadapter.jpg?alt=media\&token=6b3368e5-998f-4424-812d-e3aa1109631f) ## 2. How to works The adapter and the inbound service are linked together as shown in the figure below. ![](https://1242145269-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FnC2042spZ2TIbqdRA37S%2Fuploads%2FYoKc5mydUNHr7eB8ZwYT%2Fadapter-internal.jpg?alt=media\&token=3d8311b7-8416-4d49-8a5e-840efef974e3) Similarly, the adapter and the outbound service are linked in the following fashion. ![](https://1242145269-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FnC2042spZ2TIbqdRA37S%2Fuploads%2FQWY1sJ0rJ8CuDM3ozg0f%2Foutbound.jpeg?alt=media\&token=198e778e-26e7-4cb0-b5a9-671a576a89ed) ## 3. List of Existing Adapters * Gupshup-Whatsapp * Netcore-Whatsapp * UCI Web Channel * Firebase Adapter ## 4. How to create an adapter [Click here](/use/developer/contribution-guide/create-an-adapter) to see how to create your own adapter. # Firebase Notification Adapter ## Firebase Adapter ### 1. Overview UCI provides a firebase adapter which sends push notifications to your Mobile or Web based application. ### 2. How to send firebase notification ? We need a mobile or web application that will register the FCM tokens on firebase. These tokens can be later used to trigger firebase notification. We can trigger these notification in two ways. #### 1. Use UCI Internal Apis We can store these registered tokens in UCI & use UCI fetch api to get these tokens for notification. Follow below steps to do this. 1. Register FCM tokens on any mobile or web application using the [firebase details of UCI](#3.-contact-the-administrator). Add this FCM token on UCI using below API: ``` curl --location --request POST 'http://[HOST]:[PORT]/fusionAuth/registerUserFcmToken' \ --header 'Content-Type: application/json' \ --data-raw '{ "device": { "id":"cytMfcuBgpembxn7GfNvDh:APA91bGhiETdrh7Kq5ERG1XTQ6k53p9pVZOnMOmMFIVJVy_h4rwFb03F9AWsJ9Y51-Pc_DCt1kppHhw976WjTA_mh6WZO1pBll2A2DrUNFvG6uOjpQhc1WQoFtQUvxc3ka9e_TbBriu8", "type":"fcm" }, "users": [ { "registrationChannel": "uci-web-channel", "username": "john", "mobilePhone": "7597185712" } ] }' ``` 2. The tokens registered can be fetched via the [API](https://github.com/samagra-comms/community/blob/main/use/developer/uci-basics/adapters/http:/\[HOST]:\[PORT]/fusionAuth/fetchFcmTokens/README.md). 3. Create a new service [here](https://github.com/samagra-comms/community/blob/main/use/developer/uci-basics/adapters/http:/\[HOST]:\[PORT]/console/data/default/schema/public/tables/service/browse/README.md) with the fetch token API in config. 4. Create a new user segment [here](https://github.com/samagra-comms/community/blob/main/use/developer/uci-basics/adapters/http:/\[HOST]:\[PORT]/console/data/default/schema/public/tables/userSegment/browse/README.md) with the service created in step 3. Or you can use an already created user segment for this case. `cf085492-266f-46d1-841c-1b38e93bce3f`. 5. Create a new conversation logic with below logic ``` curl --location --request POST 'http://[HOST]:[PORT]/admin/v1/conversationLogic/create' \ --header 'admin-token: EXnYOvDx4KFqcQkdXqI38MHgFvnJcxMS' \ --header 'Content-Type: application/json' \ --data-raw '{ "data": { "name": "Firebase Broadcast Logic", "transformers": [ { "id": "774cd134-6657-4688-85f6-6338e2323dde", "meta": { "body": "Hello ${name}-${phoneNo}, Test Notification", "type": "JS_TEMPLATE_LITERALS", "params": [ "name", "phoneNo" ] }, "type": "broadcast" } ], "adapter": "2a704e82-132e-41f2-9746-83e74550d2ea" } }' ``` 6. Create a new bot with the conversation logic from step 5, user segment from step 4. ``` curl --location --request POST 'http://[HOST]:[PORT]/admin/v1/bot/create' \ --header 'admin-token: EXnYOvDx4KFqcQkdXqI38MHgFvnJcxMS' \ --header 'Content-Type: application/json' \ --data-raw '{ "data": { "startingMessage": "Test Firebase Notification", "name": "Test Firebase Notification Bot", "users": ["cf085492-266f-46d1-841c-1b38e93bce3f"], "logic": [ "03ec1ac2-76b3-496a-8184-9e9e9d01c048" ], "status": "enabled", "startDate": "2022-06-13", "endDate": "2023-06-13" } }' ``` The table below explains all the request keys while creating a conversation logic along with their description. | Key | Type | Required | Description | | ------------------------------------------------------ | ------ | -------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | data | object | Yes | The root object of the data. | | data.name | string | Yes | The name of the [adapter](/use/developer/uci-basics/adapters). | | data.description | string | Yes | The description of adapter. | | data.transformers | array | Yes | An array of transformers. | | data.transformers.id | string | Yes | The ID of the [transformer](/use/developer/uci-basics/transformers) returned by UCI when this particular transformer was created. | | data.transformers.meta | object | Yes | An object containing metadata for the transformer. | | data.transformers.meta.form | string | Yes | The URL of the form used by the transformer where the form is hosted. | | data.transformers.meta.formID | string | Yes | The ID of the ODK form used by the transformer. | | data.transformers.meta.title | string | Yes | The title of the transformer. | | data.transformers.meta.body | string | Yes | The body of the transformer. | | data.transformers.meta.serviceClass | string | Yes | The service class of the transformer. Used for prefilling in ODK responses. Currently only `SurveyService` is supported as a valid value. | | data.transformers.meta.hiddenFields | array | No | An array of hidden fields in the transformer. Hidden fields are used to prefill data in form, for example, mobile name. | | data.transformers.meta.hiddenFields.name | string | Yes | Name of the field that is used as a placeholder in the form XML. | | data.transformers.meta.hiddenFields.path | string | Yes | Path of the object from which the data needs to be filled in. For example if path is `mobilePhone`, it would be filled with the data stored in `mobilePhone` field in the object provided by config field. | | data.transformers.meta.hiddenFields.type | string | Yes | Currently the only type supported is `param`. | | data.transformers.meta.hiddenFields.config | object | Yes | An object containing the data that has to be passed for injecting in the prefilled fields. An example value would be `user` which would make the `user` object data, present in UCI session, available for use. In our example `mobileNumber` is extracted from this data. | | data.transformers.meta.hiddenFields.config.dataObjName | string | Yes | The name of the data object that needs to be used in config. | | data.transformers.meta.templateType | string | Yes | The template type of the transformer. This governs how literals are treated in the template. Possible values include `JS_TEMPLATE_LITERALS` and `PYTHON_TEMPLATE_LITERALS`. | | data.adapter | string | Yes | The ID of the adapter that is returned by UCI when a particular adapter was [created](/use/developer/uci-basics/adapters#4-how-to-create-an-adapter). | 7. Hit the below api using the created bot id to send firebase notification to all registered FCM tokens. ``` curl --location --request GET 'http://[HOST]:[PORT]/campaign/start?campaignId=1ea5346d-8d98-4cf4-a470-9c234476f3d1' ``` #### 2. Use your own FCM token api We can also use third party apis that have registered FCM tokens in a specific format. For this we will also need the firebase project service key to send notifications. 1. Create an api that provides the FCM tokens in below format. ``` { "data": [ { "fcmToken": "cytMfcuBgpembxn7GfNvDh:APA91bHsaJioCNAoWAZql1lxe4szwd74CJsPEVp1ngSCrzMcft6kB9ZrZnUZ9PUVU47kGiVSUAk70ayF55nzi8vU6jlWI4AGLkTo9ZBZnwMll0ZqVKioAQARvgE4GTUwIoTWAqVUYGMN", "phoneNo": "75********", "name": "sim", "fcmClickActionUrl": "http://www.example.com/sim" }, { "fcmToken": "cytMfcuBgpembxn7GfNvDh:APA91bHsaJioCNAoWAZql1lxe4szwd74CJsPEVp1ngSCrzMcft6kB9ZrZnUZ9PUVU47kGiVSUAk70ayF55nzi8vU6jlWI4AGLkTo9ZBZnwMll0ZqVKioAQARvgE4GTUwIoTWAqVUASDF", "phoneNo": "76********", "name": "john", "fcmClickActionUrl": "http://www.example.com/john" } ] } ``` **Note:** Param **fcmClickActionUrl** is being used to mention the url to open on click ot fcm notification. This url should have the same domain as the one on which the fcm tokens are registered, else the url will not work. 2. Add Firebase service key in vault using below APIs. 1. Login API\\ ``` curl --location --request POST 'http://auth.samagra.io:9011/api/login' \ --header 'Authorization: YFpyHxhW0-NoKRwQrXgCU5QIAQq8nBNhE--i5_n3pTU' \ --header 'Content-Type: application/json' \ --header 'Cookie: refresh_token=I2RhB1AE8Ve0gpCDh4DF3ddGBvGkwuzmokIHwi-VGbZFKdo_XSTV7A' \ --data-raw '{ "loginId": "loginId", "password": "password", "applicationId": "a1313380-069d-4f4f-8dcb-0d0e717f6a6b" }' ``` \ Please [contact the administrator](/use/contact-the-administrator) to get the **loginId** and **password** for this API.\ \ This API will give below response. Use the token from the response in "Add secret API".\\ ``` { "refreshToken": "-oYek_AgrkNGAP2cDTwgUkAsV72FDA-bRtbzmUS9Y-DOligCTx8FvA", "token": "eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCIsImtpZCI6IjZoMng4Ty1IMl9XWWt5dG41ZTNuMzlwUEZlWSJ9.eyJhdWQiOiJhMTMxMzM4MC0wNjlkLTRmNGYtOGRjYi0wZDBlNzE3ZjZhNmIiLCJleHAiOjE2NTU4MTcyMDYsImlhdCI6MTY1NTgxMzYwNiwiaXNzIjoiYWNtZS5jb20iLCJzdWIiOiI4ZjdlZTg2MC0wMTYzLTQyMjktOWQyYS0wMWNlZjUzMTQ1YmEiLCJqdGkiOiI0OGY3ZjQ3ZS1lMGM4LTQ4NmItYjVlNC00NDNlZTVkMjc1NGEiLCJhdXRoZW50aWNhdGlvblR5cGUiOiJQQVNTV09SRCIsInByZWZlcnJlZF91c2VybmFtZSI6InVjaS11c2VyIiwiYXBwbGljYXRpb25JZCI6ImExMzEzMzgwLTA2OWQtNGY0Zi04ZGNiLTBkMGU3MTdmNmE2YiIsInJvbGVzIjpbIm9yZ093bmVyIl0sIm93bmVyT3JnSWQiOiJ1Y2ktdXNlciIsIm93bmVySWQiOiJ1Y2ktdXNlciJ9.GOwVGcbjVSXZpswoVtkwdf8Kv48PuNbquFPdkQgmfabUOyMwzo_n83JAq82r2x5S8lVg07ZAovBEB1D2Ak2Vfmhr_N0gxL3bdq8YrT_e3P37mVY-OPGHbSdwzIoYEdM9njyttGjue9UI_I_i4oUI7ENVWv9GFzJmwEaVjhaZiE0NHcV5gmic62cCIQo8N-lzd4lMdZEQnDS_AxYCvjhsw2P4MxXNqzkK8bhqEcMWosfLaBrO_WJ4eh_Py0HuTMeeTI0uzSEyKPUSoQJZHuTTfks_VnayqwnpSKYPSx_riY_pHjTuB8S1L52P3fFRlwQPxPcKgyIu26DVDptqNmqAXw", "user": { "active": true, "registrations": [ { "applicationId": "a1313380-069d-4f4f-8dcb-0d0e717f6a6b", "id": "76179ff7-cd65-4c6f-9a95-d01720eacc75", "roles": [ "orgOwner" ], "username": "uci-user", "usernameStatus": "ACTIVE", "verified": true } ], "username": "loginId", "usernameStatus": "ACTIVE", "verified": true } } ``` \\ 2. Add a secret API\\ ```` curl --location -g --request POST 'http://[HOST]:[PORT]/admin/secret' \ --header 'ownerId: 8f7ee860-0163-4229-9d2a-01cef53145ba' \ --header 'ownerOrgId: org1' \ --header 'Authorization: Bearer loginToken' \ --header 'asset: secret' \ --header 'Content-Type: application/json' \ --data-raw '{ "secretBody": { "serviceKey": "BBHJGiL4qhQ:APA91bHfQkDIXbBbChCA3AUo5Wx9eRrAE2RWjtkLBCMxOJGmQGrUXqezKBy54xGJDegR6dM8H39r6XkSVDUOQUZ0QO9-Q_vexAM9UDCvCzZnerh8k1dIFUdIaQKdP8cRCW5KJG3TB123" }, "type": "Headers", "variableName": "uci-firebase-notification" }' ``` ```` \ This API will add the service key in vault. We will later use this secret variable name in adapter API to fetch its key for notification trigger. 3. Create a new adapter with credentials variable(Eg. uci-firebase-notification) added in step 2. \\ ``` curl --location --request POST 'http://[HOST]:[PORT]/admin/v1/adapter/create' \ --header 'admin-token: EXnYOvDx4KFqcQkdXqI38MHgFvnJcxMS' \ --header 'Content-Type: application/json' \ --data-raw '{ "data": { "name": "UCI Firebase Web Notification", "channel": "web", "provider": "firebase", "config": { "credentials": { "vault": "samagra", "variable": "uci-firebase-notification" } } } }' ``` \ This API will create a new adapter and will return the adapter id (Eg. 2a704e82-132e-41f2-9746-83e74550d2ea). We will use this adapter id later in conversation logic. 4. Create a new service [here](https://github.com/samagra-comms/community/blob/main/use/developer/uci-basics/adapters/http:/\[HOST]:\[PORT]/console/data/default/schema/public/tables/service/browse/README.md) with the your token API. 5. Create a new user segment [here](https://github.com/samagra-comms/community/blob/main/use/developer/uci-basics/adapters/http:/\[HOST]:\[PORT]/console/data/default/schema/public/tables/userSegment/browse/README.md) with the service created in step 4. 6. Create a new conversation logic with adapter from step 3 and below logic. ``` curl --location --request POST 'http://[HOST]:[PORT]/admin/v1/conversationLogic/create' \ --header 'admin-token: EXnYOvDx4KFqcQkdXqI38MHgFvnJcxMS' \ --header 'Content-Type: application/json' \ --data-raw '{ "data": { "name": "Firebase Broadcast Logic", "transformers": [ { "id": "774cd134-6657-4688-85f6-6338e2323dde", "meta": { "body": "Hello ${name}-${phoneNo}, Test Notification", "type": "JS_TEMPLATE_LITERALS", "params": [ "name", "phoneNo" ] }, "type": "broadcast" } ], "adapter": "2a704e82-132e-41f2-9746-83e74550d2ea" } }' ``` 7. Create a new bot with the conversation logic from step 6, user segment from step 5. ``` curl --location --request POST 'http://[HOST]:[PORT]/admin/v1/bot/create' \ --header 'admin-token: EXnYOvDx4KFqcQkdXqI38MHgFvnJcxMS' \ --header 'Content-Type: application/json' \ --data-raw '{ "data": { "startingMessage": "Test Firebase Notification", "name": "Test Firebase Notification Bot", "users": ["cf085492-266f-46d1-841c-1b38e93bce3f"], "logic": [ "03ec1ac2-76b3-496a-8184-9e9e9d01c048" ], "status": "enabled", "startDate": "2022-06-13", "endDate": "2023-06-13" } }' ``` 8. Hit the below api using the created bot id to send firebase notification to all registered FCM tokens. ``` curl --location --request GET 'http://[HOST]:[PORT]/campaign/start?campaignId=1ea5346d-8d98-4cf4-a470-9c234476f3d1' ``` ### 3. Message Receipts Api We have two apis that provides the option to add delivery & read receipts for the received message. When a firebase notification is sent, some extra data required for the receipt apis are also sent in it. **externalId** (Message ID), **destAdd** (User ID) & **fcmDestAdd** (User FCM Token) will be sent in the notification data to be later used in the receipt apis. **3.1. Delivery Receipt** This api stores the delivery receipt against a message. To sent a delivery report use the below curl request with required details. ```` ``` curl --location --request POST 'http://[HOST]:[PORT]/firebase/web' \ --header 'Content-Type: application/json' \ --data-raw '{ "text": "", "from": "", "messageId": "", "eventType": "DELIVERED", "report": { "externalId": "70eb40a7eb4b4f10890b3f2f865e307e", "destAdd": "75********", "fcmDestAdd": "cytMfcuBgpembxn7GfNvDh:APA91bHsaJioCNAoWAZql1lxe4szwd74CJsPEVp1ngSCrzMcft6kB9ZrZnUZ9PUVU47kGiVSUAk70ayF55nzi8vU6jlWI4AGLkTo9ZBZnwMll0ZqVKioAQARvgE4GTUwIoTWAqASDFGH" } }' ``` ```` **3.2. Read Receipt** This api stores the read receipt against a message. To sent a read report use the below curl request with required details. ```` ``` curl --location --request POST 'http://[HOST]:[PORT]/firebase/web' \ --header 'Content-Type: application/json' \ --data-raw '{ "text": "", "from": "", "messageId": "", "eventType": "READ", "report": { "externalId": "70eb40a7eb4b4f10890b3f2f865e307e", "destAdd": "75********", "fcmDestAdd": "cytMfcuBgpembxn7GfNvDh:APA91bHsaJioCNAoWAZql1lxe4szwd74CJsPEVp1ngSCrzMcft6kB9ZrZnUZ9PUVU47kGiVSUAk70ayF55nzi8vU6jlWI4AGLkTo9ZBZnwMll0ZqVKioAQARvgE4GTUwIoTWAqASDFGH" } }' ``` ```` ### 4. History Api UCI provides an api to fetch the conversation history against a bot or a user. **4.1. User History** Use below API to fetch the history against a bot. \*\*\*\* This API uses below required parameters to fetch the user history. * **userId**: Phone number of user * **provider**: Conversation provider. Eg. firebase (for in-app notification) * **start & end date**: Conversations between start & end date. ``` curl --location --request GET 'http://[HOST]:[PORT]/xmsg/history?userId=75********&provider=firebase&endDate=03-07-2022&startDate=01-07-2022' ``` **Limitation:** This API can provide maximum 1000 conversations. **4.2. Bot History** Use below API to fetch the history against a user. This API uses below required parameters to fetch the user history. * **botId**: Conversation id * **provider**: Conversation provider. Eg. firebase (for in-app notification) * **start & end date**: Conversations between start & end date. ``` curl --location --request GET 'http://[HOST]:[PORT]/xmsg/history/dump?provider=firebase&botId=1ea5346d-8d98-4cf4-a470-9c234476f3d1&endDate=19-07-2022&startDate=05-07-2022' ``` **Limitation:** This API can provide conversations for 15 days only. These APIs will give the response in below format. ``` { "timestamp": "2022-07-20T09:24:06Z", "status": 200, "error": null, "message": null, "path": "/xmsg/history", "result": { "total": 1, "records": [ { "messageId": "4e2aab66-012f-4e5b-92ff-9b6ad6c2859b", "messageState": "DELIVERED", "provider": "firebase", "botUuid": "694c3306-d27f-431c-87f4-ae37a41204a4", "payload": { "text": "Hello surabhi-75********, Test Notification" }, "channel": "web", "ownerOrgId": "ORG_001", "id": "2004a970-f902-11ec-bf0b-19880ce512a2", "sessionId": "1bc51202-2311-4831-aaee-cd481d9d916e", "ownerId": "abc51102-9814-4211-b71e-vd20vb9d8g57", "fromId": "75********", "userId": "admin", "timestamp": "2022-07-01T05:53:26.389" } ] } } ``` # User Segment Coming Soon # Schema Overview UCI is combination on UCI Core & UCI Api. These two have their own database schema. In the further document we will understand schema of these two. 1. [UCI Core Schema](/use/developer/uci-basics/schema-overview/uci-core-schema) 2. [UCI API Schema](/use/developer/uci-basics/schema-overview/uci-api-schema) # UCI Core Schema UCI Core is the backend of UCI and consists of all java repositories. These repositories uses two different schemas, one is for storing XMessages and other is for storing assessment data for ODK forms. 1. [XMessage Schema](/use/developer/uci-basics/schema-overview/uci-core-schema/xmessage-schema) 2. [Assessment DB Schema](/use/developer/uci-basics/schema-overview/uci-core-schema/assessment-db-schema) # XMessage Schema UCI saves all conversation data in form of XMessages. These XMessages follow a certain [specification](/use/developer/uci-basics/xmessage-specification). These XMessage will be saved in cassandra DB. Its object is defined in [dao repository](https://uci.sunbird.org/use/developer/uci-basics/schema-overview/uci-core-schema/pages/mN86WTnKfapEwuzEiaut#2.3.-dao). ## Schema <table><thead><tr><th width="184.43902439024393">Field</th><th width="150">Type</th><th>Description</th></tr></thead><tbody><tr><td>id</td><td>uuid</td><td>unique identification</td></tr><tr><td>userId</td><td>text</td><td>to - phone of user/admin</td></tr><tr><td>fromId</td><td>text</td><td>from - phone of user/admin</td></tr><tr><td>channel</td><td>text</td><td>channel (Eg. Whatsapp/SMS)</td></tr><tr><td>provider</td><td>text</td><td>provider (Eg. Gupshup/Netcore)</td></tr><tr><td>timestamp</td><td>timestamp</td><td></td></tr><tr><td>messageState</td><td>text</td><td>state(Eg. Replied/Sent/Read)</td></tr><tr><td>xMessage</td><td>text</td><td>xMessage XML stringyfied</td></tr><tr><td>app</td><td>text</td><td>bot name</td></tr><tr><td>auxData</td><td>text</td><td>auxilary data</td></tr><tr><td>messageId</td><td>text</td><td>id of message from provider</td></tr><tr><td>replyId</td><td>text</td><td>id of user to reply to</td></tr><tr><td>causeId</td><td>text</td><td></td></tr><tr><td>sessionId</td><td>uuid</td><td>session id of conversation</td></tr><tr><td>ownerOrgId</td><td>text</td><td>owner org id of bot</td></tr><tr><td>ownerId</td><td>text</td><td>owner id</td></tr><tr><td>botUuid</td><td>uuid</td><td>bot uuid</td></tr></tbody></table> # Assessment DB Schema UCI uses postgreSQL for storing data of conversations. ODK transformer uses this database to store its assessment data for each question & answer. ## Schema Assessment data consists of 4 tables, which are listed below. ### Question This table keeps the details of question asked to user. | Field | Type | Description | | -------------- | ------------ | ------------------------------- | | id | uuid | | | form\_id | varchar(100) | odk form id | | form\_version | varchar(10) | odk form version | | x\_path | varchar(500) | odk question xpath | | question\_type | varchar(500) | | | meta | jsonb | extra meta data in form of json | | created | timestamp | | | updated | timestamp | | ### Assessment This table keeps the details of answer submitted by user for a specific question. | Field | Type | Description | | ---------- | --------- | --------------------------------- | | id | uuid | | | question | uuid | id of table question | | answer | text | answer given by user for question | | bot\_id | uuid | bot id | | user\_id | uuid | fusion auth returned user id | | device\_id | uuid | fusion auth returned user id | | meta | jsondb | extra meta data in form of json | | created | timestamp | | | updated | timestamp | | ### Xmessage This table keeps the records of messages sent to user against his/her phone number. | Field | Type | Description | | ----------------- | ------------- | -------------------- | | id | bigint | | | phone\_no | varchar(15) | phone number of user | | updated\_at | timestamp | | | message | varchar(1000) | message sent to user | | is\_last\_message | boolean | | ### XMessage State This table keeps the odk instance xml state against the user phone number and form name. | Field | Type | Description | | --------------- | ------------ | ------------------------- | | id | bigint | | | phone\_no | varchar(15) | phone number of user | | state | text | current xml state of form | | previous\_path | varchar(100) | previous question xpath | | bot\_form\_name | varchar(50) | odk form name | | updated\_at | timestamp | | # UCI API Schema UCI API schema is keeps the information regarding the bot and its configuration. When a new bot is created via [APIs](/use/developer/api-documentation/bot-setup-apis) its data will be stored in this Database. ## Schema ### Adapter This table keeps the adapter information. These adapter will be associated to the [conversation logic](https://uci.sunbird.org/use/developer/uci-basics/schema-overview/pages/iyh1Cgn2i7ue92G4ym2v#2.3-add-a-conversation-logic) to determine the adapter to be used while having the conversation. API to add a new adapter is given [here](https://uci.sunbird.org/use/developer/uci-basics/schema-overview/pages/iyh1Cgn2i7ue92G4ym2v#2.1-add-adapter). | Field | Type | Description | | ----------- | --------- | --------------------------------------------------- | | id | uuid | | | channel | text | Eg. whatsapp/sms | | provider | text | Eg. netcore/gupshup | | config | jsonb | json configuration for adapter, Eg. its credentials | | name | text | adapter name | | created\_at | timestamp | | | updated\_at | timestamp | | ### Transformer This table store the transformers info and configuration, which are already implemented and working in UCI. [Click here](/use/developer/uci-basics/transformers) to know what are transformers & how they work. | Field | Type | Description | | ----------- | --------- | ---------------------------- | | id | uuid | | | name | text | | | tags | text\[] | | | config | jsonb | json configuration | | service\_id | uuid | foreign key of service table | | created\_at | timestamp | | | updated\_at | timestamp | | ### Service This table store the info to use when fetching the users data. The service will be added with user segment, which later will be added in the bot table when creating [one](https://uci.sunbird.org/use/developer/uci-basics/schema-overview/pages/iyh1Cgn2i7ue92G4ym2v#2.4-add-a-bot). | Field | Type | Description | | ----------- | --------- | ------------------------------------------------------------------------------ | | id | uuid | | | type | text | type of service Eg. gql/get | | config | jsonb | json configuration for fetching data from, Eg. get url/gql query configuration | | name | text | | | created\_at | timestamp | | | updated\_at | timestamp | | ### User Segment This table store the details to fetch the users data for different conditions like all, by id or by phone number. | Field | Type | Description | | ----------- | --------- | ------------------------------------------------------------------------------------ | | id | uuid | | | name | text | | | all | uuid | service table id to be used when fetching the all users data | | byId | uuid | service table id to be used when fetching the users data for a specific id | | byPhone | uuid | service table id to be used when fetching the users data for a specific phone number | | created\_at | timestamp | | | updated\_at | timestamp | | ### Conversation Logic This table stores the bot related logic like form id or message body. [Click here](https://uci.sunbird.org/use/developer/uci-basics/schema-overview/pages/iyh1Cgn2i7ue92G4ym2v#2.3-add-a-conversation-logic) to see the API to add conversation logic. | Field | Type | Description | | ------------ | --------- | --------------------------------- | | id | uuid | | | name | text | | | transformers | jsondb | json array of transformers config | | adapter | uuid | id of adapter table | | description | text | | | created\_at | timestamp | | | updated\_at | timestamp | | ### Bot This table keeps the details of bot name, starting message, [logic](#conversation-logic) to use, [user segment](#user-segment) to use etc. API to add a bot can be found [here](https://uci.sunbird.org/use/developer/uci-basics/schema-overview/pages/iyh1Cgn2i7ue92G4ym2v#2.4-add-a-bot). | Field | Type | Description | | ---------------- | ------- | ---------------------------------------------- | | id | uuid | | | name | text | bot name for identification | | starting message | text | a message to start a conversation on a channel | | users | text\[] | array of user segment ids | | logicIDs | text\[] | array of conversation logic ids | | status | text | Eg. enabled/draft | | start\_date | date | conversation activation date | | end\_date | date | conversation de-activation date | | ownerId | text | id of owner of this bot | | ownerOrgId | text | org id of owner of this bot | | purpose | text | purpose of the bot creation | ### # Development environment ## Development environment [Backend Setup](/use/developer/development-environment/backend-setup) [Setting up IDE](/use/developer/development-environment/setting-up-ide) [Environment variable setup](/use/developer/development-environment/environment-variable-setup) [Debug services](/use/developer/development-environment/debug-services) [Build and Execute UCI](/use/developer/development-environment/build-and-execute-uci) [Frontend Setup (Admin Console)](/use/developer/development-environment/frontend-setup-admin-console) [Frontend Setup (PWA)](/use/developer/development-environment/frontend-setup-pwa) # Backend Setup ### Getting Started ### 1. Introduction The Unified Communications Interface (UCI) aims to democratize the use of different communication channels such as WhatsApp, Telegram, SMS, email and more for governance use cases through a standard configurable manner that is reusable and scalable across all governance use cases. ### 2. Overview This Document help you to Setup UCI (Unified Communications Interface) Project and test APIs on your local machine. ### 3. Prerequisite 1. Install Git ``` $ sudo apt-get update $ sudo apt-get upgrade $ sudo apt-get install git ``` 1. Install Java 11 ``` $ sudo apt-get install openjdk-11-jdk ``` 1. Install Docker * install docker using [docker](https://docs.docker.com/engine/install/ubuntu/) installation guide. 2. Install Cassandra * install cassandra using [cassandra](https://cassandra.apache.org/doc/latest/cassandra/getting_started/installing.html) installation guide. 3. Install Maven ``` $ sudo apt-get install maven ``` 1. Install IDE ([STS](https://spring.io/tools) / [Eclipse](https://www.eclipse.org/downloads/) / [IntelliJ](https://www.jetbrains.com/idea/download/)) 2. Lombok Enabled * Enable [lombok](https://www.baeldung.com/lombok-ide) in Eclipse / STS / intelliJ. 3. Install [Postman](https://www.postman.com/downloads/) (for testing API) 4. Install Kafka, Zookeeper * install kafka using [kafka](https://www.onlinetutorialspoint.com/kafka/how-to-install-apache-kafka-on-ubuntu-18-04.html) installation guide. 5. Install Redis * Install Redis using installation [guide](https://www.digitalocean.com/community/tutorials/how-to-install-and-secure-redis-on-ubuntu-18-04). 1. Install Postgresql * Setup PostgreSQL using [Quickstart](https://www.postgresql.org/download/linux/ubuntu/) guide. ### 4. Setup #### 4.1 For first time 1. Fork following repositories ``` https://github.com/samagra-comms/dao https://github.com/samagra-comms/utils https://github.com/samagra-comms/message-rosa https://github.com/samagra-comms/adapter https://github.com/samagra-comms/outbound https://github.com/samagra-comms/orchestrator https://github.com/samagra-comms/inbound https://github.com/samagra-comms/transformer https://github.com/samagra-comms/broadcast-transformer ``` 1. Clone all forked repositories ``` git clone repository-link ``` 1. Import all cloned repositories into IDE and checkout to the latest branch Eg. release-4.9.0 2. If project is not build by default then build using ``` $ mvn clean install -U ``` 1. Add projects in IDE [link](/use/developer/development-environment/setting-up-ide) 2. Add [Enviorment Variable](broken://pages/kylVK0adlYVaxFg8SRqb) in IDE. 3. We can run the services via IDE. if you want to run these via terminal or docker use this [link](/use/developer/development-environment/build-and-execute-uci) #### 4.2 Routine 1. Run spring boot application for following projects * Inbound * Orchestrator * Transformer * Broadcast Transformer * Outbound 2. To debug these service, we should follow the same routine. [Click here](/use/developer/development-environment/debug-services) to check the detailed overview of debugging services. ### 5. APIs #### 5.1 For first time 1. Import following files to postman : * Samagra Inbound - [Apis.postman\_collection](https://github.com/samagra-comms/community/blob/main/media/Inbound-Sharing.postman_collection.json) * Samagra Transformer - [Apis.postman\_collection](https://github.com/samagra-comms/community/blob/main/media/Transformer-Sharing.postman_collection.json) 2. create new enviorment **inbound** with properties : | Variable | Type | Initial Value | Current Value | | :------: | :-----: | :------------: | :------------: | | baseUrl | default | localhost:8085 | localhost:8085 | 3. create new enviorment **transformer** with following properties : | Variable | Type | Initial Value | Current Value | | :------: | :-----: | :------------: | :------------: | | baseUrl | default | localhost:9091 | localhost:9091 | #### 5.2 Routine : 1. Test bot APIs with : ``` Collections . └── Samagra Inbound - Apis └── Bot - Messages └── ... ``` # Setting up IDE ### 1. Overview In this doc we'll configure an IDE (Eclipse/IntelliJ) for setting up development enviorment for UCI. ### 2. Setting up IntelliJ #### 2.1 Importing Projects * **Import new project** ``` File -> New -> Project from existing source ``` Then open project as **maven project** in IntelliJ. * For importing multiple projects in intelliJ click on **maven** (in right toolbar) and click on **+** for opening multiple projects. ![add multiple projects](https://1242145269-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FnC2042spZ2TIbqdRA37S%2Fuploads%2Fgit-blob-9951469f93986945ef62496df2078cf01f73724e%2Fadd_multiple_projects.png?alt=media) * After importing all the projects in your IntelliJ, now reload all maven project for first time setup. ![reload projects](https://1242145269-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FnC2042spZ2TIbqdRA37S%2Fuploads%2Fgit-blob-c01859d26ed01b0570b2758414829b913fa457a8%2Freload_all_maven_projects.png?alt=media) #### 2.2 Setting configurations For setting configuration for any project follow below steps * Goto edit configuration. ![edit config](https://1242145269-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FnC2042spZ2TIbqdRA37S%2Fuploads%2Fgit-blob-4fbe199243e56a6a8e667ddda8c5306ac71f9aeb%2Fedit_configuration.png?alt=media) * Add new configuration of type Application. ![new conf type application](https://1242145269-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FnC2042spZ2TIbqdRA37S%2Fuploads%2Fgit-blob-bd08d26c8afe5cb9646facadf2aefb406856b38b%2Ftype_application.png?alt=media) * Now give name to config, select module, select JRE, give path of main class of that module, select working directory as shown in picture. ![define properties](https://1242145269-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FnC2042spZ2TIbqdRA37S%2Fuploads%2Fgit-blob-041daa4a17c7f64fa52bff584548c3680fe94df9%2Fadd_name_and_module.png?alt=media) * For handling enviorment variables, click on edit enviorment variables (if this colum not shown by default, enable it from **modify-options**). ![enviorment variables](https://1242145269-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FnC2042spZ2TIbqdRA37S%2Fuploads%2Fgit-blob-214c574f5f8cb88c26f018d695951f5f993547e2%2Fenv_variable.png?alt=media) * Make these configuration for following projects\ inbound\ orchestrator\ transformer\ outbound #### 2.3 Build and Run Now we can build and Run the projects using below steps * Required Plugins to build the project\ maven\ docker\ lombok * Now simply select configuration and click on Run(Shift+F10), to run the project. ### 3. Setting up Eclipse #### 3.1 Importing Projects * **Import Projects into Eclipse** ``` File -> Import -> Projects From Git -> Existing Local Repository -> Select You project -> Finish ``` Import all the project like this. * After Importing, Update all project for first time setup. ``` Project -> Update Maven Project -> Select All Projects -> Update ``` ![update all](https://1242145269-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FnC2042spZ2TIbqdRA37S%2Fuploads%2Fgit-blob-08874b12ad201ea53fcd69c394411c18e5c283bb%2Fsts_update_all.png?alt=media) #### 3.2 Setting Configurations For setting configuration for any project follow below steps * Make new Configuration for spring-boot-app ``` Run -> Run Configurations -> Spring Boot App ``` * Now give name to config, select project, select main class of project and click Apply. ![](https://1242145269-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FnC2042spZ2TIbqdRA37S%2Fuploads%2Fgit-blob-9e0030c2dc05fa79c29098b108603d962693f10b%2Fsts_name_config.png?alt=media) ![](https://1242145269-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FnC2042spZ2TIbqdRA37S%2Fuploads%2Fgit-blob-9e0030c2dc05fa79c29098b108603d962693f10b%2Fsts_name_config.png?alt=media) * Now for handling Enviorment Variables click on Enviorment in config window.\ Here we can put enviorment variable's value. ![](https://1242145269-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FnC2042spZ2TIbqdRA37S%2Fuploads%2Fgit-blob-ceb21b8c29b80730330b99702f0d6d57d50abc3d%2Fsts_env_variable.png?alt=media) * Make these configuration for following projects :\ inbound\ orchestrator\ transformer\ outbound #### 3.3 Build and Run : To build and Run in Eclipse, Simply Run the project as Spring Boot App. ``` Run -> Run As -> Spring Boot App ``` # Environment variable setup # Debug services ### 1. Overview A service can be debugged via a IDE tool if you are using one to run the services or via checking the docker conatiner logs if you are using the docker image to run it. ### 2. Order of debugging services To debug any issue or error, follow below order. We should check the service in this order as the message processes in this order only. 1. Inbound 2. Orchestrator 3. Transformer 4. Outbound A message will be first received in the inbound service to convert the message to XMessage format then it will be sent to the kafka topic which is being listened by the orchestrator service. To check the build/execution flow of the services [click chere](broken://pages/n9210QJpaXYg3krHXy1T). ### 3. Debug on IDE Tool When a service is started from a IDE tool, it will also show the logs associated with the service. If you are not receiving message reply or the service is not starting, please see the logs to check the issue, and work on the solution accordingly. You can also add debugger to the IDE tool to add break points in the code to check where the issues is generating from. If you don't know how to, check below videos for reference. * How to add debugger in [Spring boot tool](https://www.youtube.com/watch?v=w5woL3znVuA) * How to add debugger in [intellij](https://www.youtube.com/watch?v=1bCgzjatcr4) ### 4. Debug for Docker Image If you have started the services using docker images, you should follow below steps to debug them. **1. Check All Services logs** * Run command below to check logs for all the running services. `docker-compose logs --follow --tail 10` **2. Check Single Service logs** * If you are sure about the service which is giving error, we can check the logs for that particular service. First check the container id by running the command below. `docker ps` It will return a list of running containers. Each container has a container id as shown in the image. ![](https://1242145269-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FnC2042spZ2TIbqdRA37S%2Fuploads%2Fgit-blob-0eb2290e83eb1ed3443be253c2089c48e69b8e38%2Fdocker-ps.png?alt=media) * To check logs for the container, run command `docker logs --follow --tail 10 container_id` To check the logs for kafka/redis etc services, you should follow the above steps. ### **5. Common mistakes or errors** 1. Missing environment variables 2. Connection issues with kafka/cassandra/postgresql etc. 3. Port in use by another service. 4. Java version conflict # Build and Execute UCI ## 1. Overview UCI has a total of 9 repositories which are being used to process the bot messages from inbound to outbound. We have 4 main repositories who will process the messages from receiving the messages from channel(Sms/Whatsapp), converting the message to xmessage, authenticating the user, producing the next message, converting it back to message format for channel and sending it to the channel(Sms/Whatsapp). The other repositories are the helper repositories with each doing different jobs. ## 2. Build Flow The repositories should be built & then executed in the sequence mentioned below. ### **2.1. Utils** Utils communicate with the federation service via apis. It holds all the utility/helper classes which will be used in other repositories. To clone this git repository use [link](https://github.com/samagra-comms/utils). Once the repository is cloned, build it using the command below. ``` mvn clean install -DskipTests ``` ### **2.2. Message Rosa** Message Rosa holds all the core models for XMessage & its property fields. To clone this git repository use [link](https://github.com/samagra-comms/message-rosa). Once the repository is cloned, build it using the command below. ``` mvn clean install -DskipTests ``` ### **2.3. Dao** Dao holds the dao & repository classes for XMessage. To clone this git repository use [link](https://github.com/samagra-comms/dao). Once the repository is cloned, build it using the command below. ``` mvn clean install -DskipTests ``` ### **2.4. Adapter** Adapters convert information provided by channels (SMS, Whatsapp) for each specific provider to xMessages and vice versa. Adapters are gateway to the external services and are responsible for receiving user response and sending the response to users. To clone this git repository use [link](https://github.com/samagra-comms/adapter). Once the repository is cloned, build it using the command below. ``` mvn clean install -DskipTests ``` ### **2.5. Inbound** Inbound receives the messages from a channel, and uses the channel adapter to convert it to XMessage format. Once the message is converted, it will be pushed to the kafka topic, the orchestrator will listen to this topic to further process it. To clone this git repository use [link](https://github.com/samagra-comms/inbound). Once the repository is cloned, build it using the command below. ``` mvn clean install -DskipTests ``` We can use any IDE tool to run it on a local machine or we can also run this service on terminal using below command. ``` mvn spring-boot:run ``` We can also create its docker image & run the service on any server/local machine. Create docker image ``` docker build -t samagragovernance/inbound:${CURRENT_VERSION} . ``` Push docker image to docker hub ``` docker push samagragovernance/inbound:${CURRENT_VERSION} ``` Run docker image ``` docker-compose -f docker-compose.yml up -d inbound ``` ### **2.6. Orchestrator** Orchestrator authenticates & processes the user data from whom the message is received. The XMessage will then be pushed to the transformers topic based on the conversation configuration, the corresponding transformer(transformer/broadcast-transformer) will listen to this topic to further process it. To clone this git repository use [link](https://github.com/samagra-comms/orchestrator). Once the repository is cloned, build it using the command below. ``` mvn clean install -DskipTests ``` We can use any IDE tool to run it on a local machine or we can also run this service on terminal using below command. ``` mvn spring-boot:run ``` We can also create its docker image & run the service on any server/local machine. Create docker image ``` docker build -t samagragovernance/orchestrator:${CURRENT_VERSION} . ``` Push docker image to docker hub ``` docker push samagragovernance/orchestrator:${CURRENT_VERSION} ``` Run docker image ``` docker-compose -f docker-compose.yml up -d orchestrator ``` ### **2.7. Transformer** This is a ODK transformer, which transforms the previous xMessage from the user to one that needs to be sent next. It is a microservice that returns a new xMessage based on the previous user action, which will then be shown to the user. The XMessage will be pushed to the process-outbound topic, orchestrator listens to this topic and will further process it to & send it to outbound topic. To clone this git repository use [link](https://github.com/samagra-comms/transformer). Once the repository is cloned, build it using the command below. ``` mvn clean install -DskipTests ``` We can use any IDE tool to run it on a local machine or we can also run this service on terminal using below command. ``` mvn spring-boot:run ``` We can also create its docker image & run the service on any server/local machine. Create docker image ``` docker build -t samagragovernance/transformer:${CURRENT_VERSION} . ``` Push docker image to docker hub ``` docker push samagragovernance/transformer:${CURRENT_VERSION} ``` Run docker image ``` docker-compose -f docker-compose.yml up -d transformer ``` ### **2.8. Broadcast Transformer** Broadcast transformer transforms the single message to multiple messages for each user segment associated with a conversation. It is a micro service that generated multiple XMessage and push it to process-outbound topic, orchestrator listens to this topic and will further process it to & send it to outbound topic. To clone this git repository use [link](https://github.com/samagra-comms/broadcast-transformer). Once the repository is cloned, build it using the command below. ``` mvn clean install -DskipTests ``` We can use any IDE tool to run it on a local machine or we can also run this service on terminal using below command. ``` mvn spring-boot:run ``` We can also create its docker image & run the service on any server/local machine. Create docker image ``` docker build -t samagragovernance/broadcast-transformer:${CURRENT_VERSION} . ``` Push docker image to docker hub ``` docker push samagragovernance/broadcast-transformer:${CURRENT_VERSION} ``` Run docker image ``` docker-compose -f docker-compose.yml up -d broadcast-transformer ``` ### **2.9. Outbound** Outbound converts the xMessage to the one that will be sent to the channel(sms/whatsapp). It will then be sent to the network provider(Netcore/Gupshup) who will send it to the channel. To clone this git repository use [link](https://github.com/samagra-comms/outbound). Once the repository is cloned, build it using the command below. ``` mvn clean install -DskipTests ``` We can use any IDE tool to run it on a local machine or we can also run this service on terminal using below command. ``` mvn spring-boot:run ``` We can also create its docker image & run the service on any server/local machine. Create docker image ``` docker build -t samagragovernance/outbound:${CURRENT_VERSION} . ``` Push docker image to docker hub ``` docker push samagragovernance/outbound:${CURRENT_VERSION} ``` Run docker image ``` docker-compose -f docker-compose.yml up -d outbound ``` # Frontend Setup (Admin Console) ### Your machine should have yarn or npm. #### Note: Preferable versions ``` Version: node 12 npm 6 ``` ### Check the node and npm version by running the following commands. ``` node -v npm -v ``` ### Installation Steps: #### 1. Fork it You can get your own fork/copy of Frontend by using the Fork button. #### 2. Clone it You need to clone (download) it to a local machine using git clone <https://github.com/samagra-comms/uci-admin.git> Once you have cloned the uci-admin repository in GitHub, move to that folder first using the change directory command. This will change directory to a folder ``` cd uci-admin ``` Move to this folder for all other commands. #### 3. Set it up Run the following commands to see that your local copy has a reference to your forked remote repository in GitHub ``` git remote -v ``` By running the above command, you can see that the local copy has a reference to the forked remote repository in GitHub. ``` origin https://github.com/samagra-comms/uci-admin.git (fetch) origin https://github.com/samagra-comms/uci-admin.git (push) ``` #### 4. Run it #### Install dependencies ``` npm install ``` #### 5. Build ``` ng build –prod ``` Build @dist #### Run frontend application in dev environment ``` ng serve ``` Browse - <http://localhost:4000> Run standalone module in dev environment # Frontend Setup (PWA) ## Getting started ### Introduction The Unified Communications Interface (UCI) aims to democratize the use of different communication channels such as WhatsApp, Telegram, SMS, email and more for governance use cases through a standard configurable manner that is reusable and scalable across all governance use cases. Transport socket takes request from chatbot client and send it to UCI adapter to get response and return it back to bot client \[UCI] <---> \[Socket Transport Layer] <---> \[Frontend] ### Overview This document helps you to setup UCI-PWA which helps you to experience the conversation with chatbot on UCI web channel. ### Pre-requisites 1. Update to latest version ``` $ sudo apt update $ sudo apt-get update $ sudo apt-get upgrade ``` 2. Install Git ``` $ sudo apt-get install git ``` 3. Install curl ``` $ sudo apt-get install curl ``` 4. Install Node.js, npm, yarn ``` $ sudo apt install nodejs $ sudo apt install npm $ sudo npm install yarn ``` * Node version should be latest or stable (v16.15.0) * To check node version: ``` $ node -v ``` 5. Install NVM ``` $ curl -o- https://raw.githubusercontent.com/creationix/nvm/v0.39.1/install.sh | bash ``` * Make nvm command available to terminal ``` $ source ~/.nvm/nvm.sh ``` * For more information of node and NVM installation, [check here](https://www.digitalocean.com/community/tutorials/how-to-install-node-js-on-ubuntu-20-04). 6. Switching node version * Run the following command to check node version ``` node -v ``` * If your node version is not v16.15.0 or latest, even after installation, you can switch to the required version by the following command: ``` $ nvm install 16.14.0 $ nvm alias default 16.14.0 $ nvm use default ``` ## Setup ### For the first time: 1. **Fork** the repository: * To get a copy of the repository, you need to fork the following:- * [UCI-PWA](https://github.com/samagra-comms/uci-web-channel) 2. **Clone** the forked repository: * To download the forked repository, you need to clone it. ``` $ git clone repository-link ``` 3. Move to the directory with the cloned repository: ``` $ cd uci-pwa or $ cd <path of uci-pwa folder> ``` 4. Run the following command to see if your local copy has a reference to your forked remote repository in GitHub: ``` git remote -v # which should give this output- origin https://github.com/Your_Username/repo_name.git (fetch) origin https://github.com/Your_Username/repo_name.git (push) ``` 5. Import the cloned repository to the chosen IDE. 6. Install the required packages * In terminal (location should be same as the current location of the repository), run this command to install all the packages required to run the code:- ``` $ yarn install ``` 7. Setup all the environment variables:- * Make a file named '.env' in the base location of the project directory(where .env-sample is located) and add the following in it. * Contact the [administrator](#contact-the-administrator) to get the production transport socket url or setup transport socket on local using [link](https://github.com/samagra-comms/community/blob/main/use/developer/development-environment/transport-socket_pwa.md) ``` REACT_APP_TRANSPORT_SOCKET_URL=ws://url ``` * Contact the [administrator](#contact-the-administrator) to get the production inbound url or setup inbound on local using this [link](/use/developer/development-environment/backend-setup) ``` REACT_APP_INBOUND_BASE_URL=http://url ``` 8. Once all the required packages are successfully installed and .env files are setup, start the project. ## Run the project: 1. To start the project, run the following command:- ``` $ yarn start ``` 2. After running the project, browser displays the following screen: * This is the first look of UCI-PWA: ![UCI-PWA front-end first look](https://1242145269-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FnC2042spZ2TIbqdRA37S%2Fuploads%2Fgit-blob-c0169fda7dc34edbab0b89c0d3484e33c4997d5f%2Fuci-pwa_view.png?alt=media) * The textbar given below can be used to send messages. Once you send the required starting message, conversation with the bot gets started. ![UCI-PWA front-end first look](https://1242145269-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FnC2042spZ2TIbqdRA37S%2Fuploads%2Fgit-blob-0bedb19e1adf6829364b0e088ef73afb103b3dea%2Fuci-pwa_working.png?alt=media) # Transport Socket ### Getting started #### Introduction The Unified Communications Interface (UCI) aims to democratize the use of different communication channels such as WhatsApp, Telegram, SMS, email and more for governance use cases through a standard configurable manner that is reusable and scalable across all governance use cases. Transport socket takes request from chatbot client and send it to UCI adapter to get response and return it back to bot client \[UCI] <---> \[Socket Transport Layer] <---> \[Frontend] #### Overview This document helps you to setup transport socket which helps communicate between UCI backend and frontend. To experience the conversation with chatbot on UCI web channel, UCI-PWA needs to be setup from here. #### Pre-requisites 1. Update to latest version ``` $ sudo apt update $ sudo apt-get update $ sudo apt-get upgrade ``` 2. Install Git ``` $ sudo apt-get install git ``` 3. Install curl ``` $ sudo apt-get install curl ``` 4. Install Node.js, npm, yarn ``` $ sudo apt install nodejs $ sudo apt install npm $ sudo npm install yarn ``` * Node version should be latest or stable (v16.15.0) * To check node version: ``` $ node -v ``` 5. Install NVM ``` $ curl -o- https://raw.githubusercontent.com/creationix/nvm/v0.39.1/install.sh | bash ``` * Make nvm command available to terminal ``` $ source ~/.nvm/nvm.sh ``` * For more information of node and NVM installation, [check here](https://www.digitalocean.com/community/tutorials/how-to-install-node-js-on-ubuntu-20-04). 6. Switching node version * Run the following command to check node version ``` node -v ``` * If your node version is not v16.15.0 or latest, even after installation, you can switch to the required version by the following command: ``` $ nvm install 16.14.0 $ nvm alias default 16.14.0 $ nvm use default ``` 7. Set up on Docker * Install [Docker](https://www.digitalocean.com/community/tutorials/how-to-install-and-use-docker-on-ubuntu-20-04) * Setup and run redis on docker ``` $ sudo docker run --name my-first-redis -p 6379:6379 -d redis ``` ### Setup #### For the first time: 1. **Fork** the repository: * To get a copy of the repository, you need to fork the following:- * [Transport Socket](https://github.com/samagra-comms/transport-socket) 2. **Clone** the forked repository: * To download the forked repository, you need to clone it. ``` $ git clone repository-link ``` 3. Move to the directory with the cloned repository: ``` $ cd transport-socket or $ cd <path of transport-socket folder> ``` 4. Run the following command to see if your local copy has a reference to your forked remote repository in GitHub: ``` git remote -v # which should give this output- origin https://github.com/Your_Username/repo_name.git (fetch) origin https://github.com/Your_Username/repo_name.git (push) ``` 5. Import the cloned repository to the chosen IDE. 6. Move to required branch: ``` $ git checkout uci-pwa ``` 7. Install the required packages * In terminal (location should be same as the current location of the repository), run this command to install all the packages required to run the code:- ``` $ yarn install ``` 8. Setup all the environment variables:- * Make a file named '.env' in the base location of the project directory(where .env-sample is located) and add the following in it. ``` REDIS_HOST=localhost REDIS_PORT=6379 SERVER_PORT=3005 ``` * [Contact the administrator](/use/contact-the-administrator) to use the production inbound url or setup inbound repository on local using this [link](/use/developer/development-environment/backend-setup) ``` ADAPTER_URL=http://localhost:8080/pwa/web ``` 9. Once all the required packages are successfully installed and .env files are setup, start the project. ### Run the project: 1. To start the project, run the following command:- ``` $ yarn start ``` # API Documentation * [**Bot Setup APIs**](/use/developer/api-documentation/bot-setup-apis) * [**History APIs**](/use/developer/api-documentation/history-apis) * [**Direct Message APIs**](/use/developer/api-documentation/direct-message-apis) * [**Vault APIs**](/use/developer/api-documentation/vault-apis) # Bot Setup APIs ## 1. Overview UCI provides a list of apis that will help you to configure bots for conversation or broadcasting. ## 2. APIs To setup a bot and start conversation with users, the following steps need to be followed. ### 2.1 Add adapter Add a new adapter for the available channel & providers. Below is a list of available parameters. * **name:** Name of the adapter * **channel**: Messaging channel. * **provider**: Message provider * **config**: * **credentials**: * **vault**: Vault service name, Default value: samagra * **variable**: Variable name added in the vault. List of available channel and provider combinations: * provider: netcore, channel: whatsapp * provider: gupshup, channel: whatsapp * provider: gupshup, channel: sms * provider: cdac, channel: sms * provider: firebase, channel: web Note: If you need some other channel or provider, contribute a new adapter for the same. [Click here](/use/developer/contribution-guide/create-an-adapter) to see how to do this. Below is a curl request for adding a new adapter in db. This adapter will be associated with a bot to determine the channel and provider for the bot when conversing. ``` curl --location --request POST 'http://UCI_API:PORT/admin/adapter' \ --header 'admin-token: {ADMIN_TOKEN}' \ --header 'Content-Type: application/json' \ --data-raw '{ "data": { "name": "UCI Gupshup Whatsapp", "channel": "whatsapp", "provider": "gupshup", "config": { "credentials": { "vault": "samagra", "variable": "uci-gupshup-whatsapp" } } } }' ``` This API will create a new adapter and will return the adapter id (Eg. 2a704e82-132e-41f2-9746-83e74550d2ea). We can use this adapter id later in [create conversation logic](#2.4-Add-a-Conversation-Logic) api. ### 2.2 Creating a Conversation Bot A conversation bot is used to communicate with the user one on one. A chatbot uses a flow defined in an odk xml form to chat with a user. The steps for creating a simple conversation chatbot are given below. ### 2.2.1 Upload ODK Form For using ODK transformer, we will first have to upload a ODK form. Follow below steps to upload a ODK form. * Convert a ODK Excel form to XML form using [Link](https://getodk.org/xlsform/). * Here is a [Sample ODK Excel Form](https://github.com/samagra-comms/docker-deploy/blob/main/media/List-QRB-Test-Bot.xlsx) for reference. * Upload this XML from using this api. ``` curl --location 'http://UCI_API:PORT/admin/form/upload' \ --header 'admin-token: {ADMIN_TOKEN}' \ --form 'form=@"{PATH_TO_YOUR_XML}"' \ --form 'mediaFiles=@"{PATH_TO_YOUR_MEDIA_1}"' \ --form 'mediaFiles=@"{PATH_TO_YOUR_MEDIA_2}"' \ --form 'mediaFiles=@"{PATH_TO_YOUR_MEDIA_N}"' ``` **Response**: The API will return a form id. Use this form id to create conversation logic API. Form id E.g. **testing\_form** ``` { "apiId": "api.form.upload", "path": "/admin/form/upload", "msgid": "fec3421b-cc04-42e2-9ce9-e004b3ded827", "result": { "status": "UPLOADED", "data": { "formID": "testing_form" } }, "startTime": "2023-07-04T12:01:00.268Z", "method": "POST", "endTime": "2023-07-04T12:01:21.785Z" } ``` ### 2.2.2 Add a Conversation Logic For any bot we will have to specify a certain configuration which is a part of conversation logic. Below is a list of available parameters. * **name:** Conversation logic unique name * **transformers:** Array of transformers * **id:** id of transformer * **meta:** * **formID:** uploaded odk form id from [upload form api](#2.2-upload-odk-form) (For ODK transformer) * **params:** Array of parameters to be used in template \*\*\*\* message (For Broadcast transformer) * **body:** (For Broadcast transformer) * **type:** type of template (For Broadcast transformer), Default. **JS\_TEMPLATE\_LITERALS** * **type:** type of transformers (Eg. broadcast/generic) \*\*\*\* * **adapter:** id \*\*\*\* of adapter from [add adapter api](#2.1-add-adapter). Use below curl to create a conversation logic. ``` curl --location 'http://UCI_API:PORT/admin/conversationLogic' \ --header 'asset: conversationLogic' \ --header 'admin-token: dR67yAkMAqW5P9xk6DDJnfn6KbD4EJFVpmPEjuZMq44jJGcj65' \ --header 'Content-Type: application/json' \ --header 'Accept: application/json, text/plain, */*' \ --header 'ownerOrgID: org01' \ --header 'ownerID: 8f7ee860-0163-4229-9d2a-01cef53145ba' \ --data '{ "data": { "name": "TEST", "description": "TEST", "transformers": [ { "id": "774cd134-6657-4688-85f6-6338e2323dde", "meta": { "form": "https://hosted.my.form.here.com", "formID": "testing_form", "title": "TEST", "body": "TEST", "serviceClass": "SurveyService", "hiddenFields": [ { "name": "mobilePhone", "path": "mobilePhone", "type": "param", "config": { "dataObjName": "user" } } ], "templateType": "JS_TEMPLATE_LITERALS", "type": "broadcast" } } ], "adapter": "6efa8087-0939-49ab-b8e5-5676e036c17b" } }' ``` **Response**: It will return a conversation logic id, use it in create bot. Eg. **be37c9f2-7f2b-4e19-b525-33b0a2aabdd5** ``` { "apiId": "api.admin.conversationLogic", "path": "/admin/conversationLogic", "msgid": "df8edb08-f887-41cd-8f7a-604d3229dabc", "result": { "id": "be37c9f2-7f2b-4e19-b525-33b0a2aabdd5", "name": "TEST", "createdAt": "2023-07-05T05:29:57.310Z", "updatedAt": "2023-07-05T05:29:57.311Z", "description": null, "adapterId": "6efa8087-0939-49ab-b8e5-5676e036c17b" }, "startTime": "2023-07-05T05:29:57.293Z", "method": "POST", "ownerId": "8f7ee860-0163-4229-9d2a-01cef53145ba", "endTime": "2023-07-05T05:29:57.314Z" } ``` ### 2.2.3 Configure a bot After the conversation logic is defined, we can use this to create a new bot. This bot must include a starting message as we will use this to start a conversation for this bot. Below is a list of available parameters. * **startingMessage:** unique message to start conversation. * **name:** unique name of bot. * **users:** array of user segment ids, currently the first one will be used. * **logic:** array of conversation logic ids from add logic api, currently the first one will be used. * **status:** status of bot, Eg. draft/disabled/enabled. * **startDate:** date from which bot will be active. * **endDate:** date till which bot will be active. Use below curl to create a conversation logic. ``` curl --location --request POST 'http://UCI_API:PORT/admin/bot' \ --header 'admin-token: {ADMIN_TOKEN}' \ --form 'botImage=@"/path/to/file"' \ --form 'data="{ \"data\": { \"startingMessage\": \"Hi Bot\", \"name\": "UCI Demo Bot\", \"users\": [], \"logic\": [\"be37c9f2-7f2b-4e19-b525-33b0a2aabdd5\"], \"status\": \"enabled\", \"startDate\": \"2023-05-4\", \"endDate\": \"2025-12-01\" } }"' ``` Response: This api will return a bot id & other bot information. Use the starting message (Eg. Hi Bot) from here to start a conversation with a bot. ### 2.3 Creating a Broadcast Bot (Optional) ### 2.3.1 Add a User Segment User segment contains the group of users to whom the broadcast would be sent out to. Use the curl below to create a user segment. ``` curl --location 'http://UCI_API:PORT/admin/user-segment' \ --header 'asset: userSegment' \ --header 'admin-token: {ADMIN_TOKEN}' \ --header 'Content-Type: application/json' \ --header 'Accept: application/json, text/plain, */*' \ --header 'ownerOrgID: org01' \ --header 'ownerID: 8f7ee860-0163-4229-9d2a-01cef53145ba' \ --data '{ "name": "TEST_USER_SEGMENT", "all": { "type": "get", "config": { "url": "http://testSegmentUrl/segments/1/mentors?deepLink=nipunlakshya", "type": "GET", "cadence": { "perPage": 100, "retries": 5, "timeout": 60, "concurrent": true, "pagination": true, "concurrency": 10, "retries-interval": 10 }, "pageParam": "page", "credentials": {} } }, "byID": { "type": "get", "config": { "url": "http://testSegmentUrl/segments/1/mentors?deepLink=nipunlakshya://chatbot", "type": "GET", "cadence": { "perPage": 100, "retries": 5, "timeout": 60, "concurrent": true, "pagination": true, "concurrency": 10, "retries-interval": 10 }, "pageParam": "page", "credentials": {} } }, "byPhone": { "type": "get", "config": { "url": "http://testSegmentUrl/segments/1/mentors?deepLink=nipunlakshya://chatbot", "type": "GET", "cadence": { "perPage": 100, "retries": 5, "timeout": 60, "concurrent": true, "pagination": true, "concurrency": 10, "retries-interval": 10 }, "pageParam": "page", "credentials": {} } } }' ``` **Response**: It will return a response containing the id of the segment that is created. Eg. **2ddea169-2db9-442b-99f2-fc85c9ea1a02** ``` { "id": "2ddea169-2db9-442b-99f2-fc85c9ea1a02", "createdAt": "2023-07-05T06:02:16.285Z", "updatedAt": "2023-07-05T06:02:16.285Z", "name": "TEST_JUL_5", "description": null, "count": 0, "category": null, "allServiceID": "00b775e2-fae9-439f-8e9d-3231dc5aed46", "byPhoneServiceID": "a279516d-3730-4b7f-b74c-de5df0206177", "byIDServiceID": null, "botId": null } ``` ### 2.2.4 Add a Broadcast bot After the conversation logic is defined, we can use this to create a new bot. This bot must include a starting message as we will use this to start a conversation for this bot. Below is a list of available parameters. * **startingMessage:** unique message to start conversation * **name:** unique name of bot \*\*\*\* * **users:** array of user segment ids, currently the first one will be used * **logic:** array of conversation logic ids from [add logic api](#2.3-create-a-conversation-logic), currently the first one will be used * **status:** status of bot, Eg. draft/enabled * **startDate:** date from which bot will be active * **endDate:** date till which bot will be active Use below curl to create a conversation logic. ``` curl --location 'http://http//UCI_API:PORT/admin/bot' \ --header 'asset: bot' \ --header 'admin-token: dR67yAkMAqW5P9xk6DDJnfn6KbD4EJFVpmPEjuZMq44jJGcj65' \ --header 'Accept: application/json, text/plain, */*' \ --header 'ownerOrgID: org01' \ --header 'ownerID: 8f7ee860-0163-4229-9d2a-01cef53145ba' \ --form 'botImage=@"{PATH_TO_IMAGE}"' \ --form 'data="{ \"data\": { \"name\": \"TEST BROADCAST BOT\", \"description\": \"TEST\", \"purpose\": \"TEST\", \"startingMessage\": \"Hi Test Broadcast\", \"startDate\": \"2023-06-16\", \"endDate\": \"2023-06-30\", \"isBroadcastBotEnabled\": true, \"segmentId\": \"1\", \"status\": \"enabled\", \"users\": [ \"2ddea169-2db9-442b-99f2-fc85c9ea1a02\" // Get this id from user segment api response ], \"logic\": [ \"be37c9f2-7f2b-4e19-b525-33b0a2aabdd5\" // Get this id from conversation logic api response ] } }"' ``` **Response**: This api will return a bot id & other bot information. Use the starting message (Eg. **Hi Test ODK**) from here to start conversation with a bot. ``` { "apiId": "api.admin.bot", "path": "/admin/bot", "apiVersion": "v1", "msgid": "ba858dbb-8710-40e3-9fca-e8f928ff2cfc", "result": { "id": "5066c217-6ae0-4e3c-9802-8c0cce660c2b", "createdAt": "2023-07-04T12:59:48.804Z", "updatedAt": "2023-07-04T12:59:48.804Z", "name": "TEST ODK BROADCAST", "startingMessage": "Hi Test Broadcast", "ownerID": null, "ownerOrgID": null, "purpose": null, "description": null, "startDate": "2023-06-15T00:00:00.000Z", "endDate": "2025-12-01T00:00:00.000Z", "status": "ENABLED", "tags": [], "botImage": "1c8516c6-c4c3-4a37-a8dd-89109e8e0e60.png" }, "startTime": "2023-07-04T12:59:48.265Z", "method": "POST", "endTime": "2023-07-04T12:59:49.097Z" } ``` # History APIs ## 1. Overview UCI provides APIs to fetch the conversation history against a bot or a user. ## **2. APIs** ### **2.1. User History** Use below API to fetch the history against a bot. \*\*\*\* This API uses below required parameters to fetch the user history. * **userId**: Phone number of user * **provider**: Conversation provider. **Eg**. gupshup * **start & end date**: Conversations between start & end date. ``` curl --location --request GET 'http://INBOUND:PORT/xmsg/history?userId=75********&provider=gupshup&endDate=03-07-2022&startDate=01-07-2022' ``` **Limitation:** This API can provide maximum 1000 conversations. ### **2.2. Bot History** Use below API to fetch the history against a user. This API uses below required parameters to fetch the user history. * **botId**: Conversation id * **provider**: Conversation provider. **Eg**. gupshup * **start & end date**: Conversations between start & end date. ``` curl --location --request GET 'http://INBOUND:PORT/xmsg/history/dump?provider=gupshup&botId=d655cf03-1f6f-4510-acf6-d3f51b488a5e&endDate=19-07-2022&startDate=05-07-2022' ``` **Limitation:** This API can provide conversations for 15 days only. ### **2.3. Bot + User History** Use below API to fetch the history against a user for a specific bot. This API uses below required parameters to fetch the history. * **botId**: Conversation id * **userId**: Phone number of user * **provider**: Conversation provider. **Eg**. pwa * **start & end date**: Conversations between start & end date. ``` curl --location 'http://INBOUND:PORT/xmsg/conversation-history?provider=pwa&endDate=22-06-2023&startDate=15-06-2023&userId=PhoneNo&botId=BOTID' ``` **Limitation:** This API can provide maximum 1000 conversations. These APIs will give the response in below format. ``` { "timestamp": "2022-07-20T09:24:06Z", "status": 200, "error": null, "message": null, "path": "/xmsg/history", "result": { "total": 1, "records": [ { "readTimestamp": "2022-08-16T12:37:17.884", "messageState": "READ", "sentTimestamp": "2022-08-16T12:37:10", "channel": "WhatsApp", "messageId": "4707198624577024267-422975385292909743", "sessionId": "e4f96af4-d714-4d41-bde6-d8cf6280b7d3", "ownerId": "b45g6af4-v714-2341-bd89-d8cf6280b76t", "fromId": "75********", "userId": "admin", "provider": "gupshup", "botUuid": "09a370f3-712c-4be6-b6aa-6c57957fb670", "payload": { "text": "What is your name?" }, "deliveryTimestamp": "2022-08-16T12:37:13.675", "ownerOrgId": "ORG_001", "id": "2a261c40-1d60-11ed-9003-6d8d3766cc00" } ] } } ``` ### **2.4. Flow Diagram** Below is the flow diagram of how the history API works. ![Flow Diagram](https://1242145269-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FnC2042spZ2TIbqdRA37S%2Fuploads%2Fgit-blob-95f98254e6e52e554f27ec876fe9fe54741ce441%2Fconversation_history.png?alt=media) # Direct Message APIs ## 1. Overview Send direct messages from outbound service. This api used below parameters to send message to devices. * **adapterId**: Id of adapter added for provider & channel **Eg**. Gupshup-Whatsapp/Netcore-Whatsapp etc. * **to.userID:** Phone number of user * **to.deviceType**: device type of user **Eg**. PHONE * **to.meta.fcmToken**: FCM token of user for firebase notifications. It is required for firebase notifications. * **payload.text**: \*\*\*\* Text to be sent to user * **payload.media.url**: URL of image to be sent to user (In case of media messages) * **payload.media.category**: category of media **Eg**. IMAGE\_URL.\ *Available categories:* IMAGE\_URL, AUDIO\_URL, VIDEO\_URL, DOCUMENT\_URL\_ * **payload.media.caption:** Caption to be sent with media file This API will give you response in below format. If the api returns a succes response, it will include a message id in result. ``` { "timestamp": "2022-08-04T11:51:51Z", "status": 200, "error": null, "message": "Message sent.", "path": "/message/send", "result": { "messageId": "2a03363a-7752-4a19-b7f7-e3a1e1423ae8" } } ``` Below are some curl requests for sending messages. ## 2. APIs ### 2.1 Netcore Whatsapp Message * **Send text message to whatsapp number** ``` curl --location --request POST 'OUTBOUND:PORT/message/send' \ --header 'Content-Type: application/json' \ --data-raw '{ "adapterId": "44a9df72-3d7a-4ece-94c5-98cf26307323", "to": { "userID": "97********", "deviceType": "PHONE" }, "payload": { "text": "Hello there!" } }' ``` * **Send media message to whatsapp number** ``` curl --location --request POST 'OUTBOUND:PORT/message/send' \ --header 'Content-Type: application/json' \ --data-raw '{ "adapterId": "44a9df72-3d7a-4ece-94c5-98cf26307323", "to": { "userID": "97********", "deviceType": "PHONE" }, "payload": { "media": { "url": "@{IMAGE_PATH/URL}", "category": "IMAGE", "text": "Image Caption" } } }' ``` ### 2.2 Gupshup Whatsapp Message * **Send text message to whatsapp number** ``` curl --location --request POST 'OUTBOUND:PORT/message/send' \ --header 'Content-Type: application/json' \ --data-raw '{ "adapterId": "44a9df72-3d7a-4ece-94c5-98cf26307324", "to": { "userID": "97********", "deviceType": "PHONE" }, "payload": { "text": "Hello there!" } }' ``` * **Send media message to whatsapp number** ``` curl --location --request POST 'OUTBOUND:PORT/message/send' \ --header 'Content-Type: application/json' \ --data-raw '{ "adapterId": "44a9df72-3d7a-4ece-94c5-98cf26307323", "to": { "userID": "97********", "deviceType": "PHONE" }, "payload": { "media": { "url": "@{IMAGE_PATH/URL}", "category": "IMAGE", "text": "Image Caption" } } }' ``` ### 2.3 Firebase Web Message * **Send message to device** The **data** field of the payload contains all the data that the backend might want to send to the user along with the message. ``` curl --location --request POST 'http:/OUTBOUND:PORT/message/send' \ --header 'Content-Type: application/json' \ --data-raw '{ "adapterId": "21f1d315-55cf-44e3-8355-4743d6519649", "to": { "userID": "97********", "deviceType": "PHONE" }, "payload": { "text": "Testing", "data": [ { "key": "fcmToken", "value": "myFCMToken" }, { "key": "fcmClickActionUrl", "value": "nipunlakshya://chatbot" }, { "key": "botDesc", "value": "This is testing description" }, { "key": "botId", "value": "This is testing bot id" } ] } }' ``` **Note:** Param **fcmClickActionUrl** is being used to mention the url to open on click ot fcm notification. This url should have the same domain as the one on which the fcm tokens are registered, else the url will not work. ### 2.4 Gupshup SMS * **Send message to phone** ``` curl --location --request POST 'OUTBOUND:PORT/message/send' \ --header 'Content-Type: application/json' \ --data-raw '{ "adapterId": "64036edb-e763-44b1-99b8-37b6c7b292c5", "to": { "userID": "97********", "deviceType": "PHONE" }, "payload": { "text": "Hi user" } }' ``` ### 2.4 CDAC SMS * **Send message to phone** For sending message to phone number via CDAC, a **templateId** which is approved by CDAC is required and the payload text should have content of this template body. ``` curl --location --request POST 'OUTBOUND:PORT/message/send' \ --header 'Content-Type: application/json' \ --data-raw '{ "adapterId": "77164079-579a-4a70-b578-7dfe2afb677a", "to": { "userID": "97*********", "deviceType": "PHONE", "meta": { "templateId": "1007180838266527905" } }, "payload": { "text": "Hi User" } }' ``` # Vault APIs ## 1. Overview UCI provides an option to store the credentials required by the adapters in vault. ## 2. APIs To add a credentials in vault, use below apis. ### 2.1 Login API This apis provides a login token to use in the add credentials api. Below are the available parameters of api. * **loginId:** login id of fusion auth(which is connected to vault) * **password:** password of fusion auth(which is connected to vault) * **applicationId:** application registered on fusion auth(which is connected to vault) **Note**: Please [contact the administrator](/use/contact-the-administrator) to get the **loginId** and **password** for this API. Use below curl to get login token. ``` curl --location --request POST 'http://auth.samagra.io:9011/api/login' \ --header 'Authorization: YFpyHxhW0-NoKRwQrXgCU5QIAQq8nBNhE--i5_n3pTU' \ --header 'Content-Type: application/json' \ --header 'Cookie: refresh_token=I2RhB1AE8Ve0gpCDh4DF3ddGBvGkwuzmokIHwi-VGbZFKdo_XSTV7A' \ --data-raw '{ "loginId": "loginId", "password": "password", "applicationId": "a1313380-069d-4f4f-8dcb-0d0e717f6a6b" }' ``` This API will give below response. Use the token from the response in [Add secret/credentials API](#2.2-add-a-secret-api). ``` { "refreshToken": "-oYek_AgrkNGAP2cDTwgUkAsV72FDA-bRtbzmUS9Y-DOligCTx8FvA", "token": "eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCIsImtpZCI6IjZoMng4Ty1IMl9XWWt5dG41ZTNuMzlwUEZlWSJ9.eyJhdWQiOiJhMTMxMzM4MC0wNjlkLTRmNGYtOGRjYi0wZDBlNzE3ZjZhNmIiLCJleHAiOjE2NTU4MTcyMDYsImlhdCI6MTY1NTgxMzYwNiwiaXNzIjoiYWNtZS5jb20iLCJzdWIiOiI4ZjdlZTg2MC0wMTYzLTQyMjktOWQyYS0wMWNlZjUzMTQ1YmEiLCJqdGkiOiI0OGY3ZjQ3ZS1lMGM4LTQ4NmItYjVlNC00NDNlZTVkMjc1NGEiLCJhdXRoZW50aWNhdGlvblR5cGUiOiJQQVNTV09SRCIsInByZWZlcnJlZF91c2VybmFtZSI6InVjaS11c2VyIiwiYXBwbGljYXRpb25JZCI6ImExMzEzMzgwLTA2OWQtNGY0Zi04ZGNiLTBkMGU3MTdmNmE2YiIsInJvbGVzIjpbIm9yZ093bmVyIl0sIm93bmVyT3JnSWQiOiJ1Y2ktdXNlciIsIm93bmVySWQiOiJ1Y2ktdXNlciJ9.GOwVGcbjVSXZpswoVtkwdf8Kv48PuNbquFPdkQgmfabUOyMwzo_n83JAq82r2x5S8lVg07ZAovBEB1D2Ak2Vfmhr_N0gxL3bdq8YrT_e3P37mVY-OPGHbSdwzIoYEdM9njyttGjue9UI_I_i4oUI7ENVWv9GFzJmwEaVjhaZiE0NHcV5gmic62cCIQo8N-lzd4lMdZEQnDS_AxYCvjhsw2P4MxXNqzkK8bhqEcMWosfLaBrO_WJ4eh_Py0HuTMeeTI0uzSEyKPUSoQJZHuTTfks_VnayqwnpSKYPSx_riY_pHjTuB8S1L52P3fFRlwQPxPcKgyIu26DVDptqNmqAXw", "user": { "active": true, "registrations": [ { "applicationId": "a1313380-069d-4f4f-8dcb-0d0e717f6a6b", "id": "76179ff7-cd65-4c6f-9a95-d01720eacc75", "roles": [ "orgOwner" ], "username": "uci-user", "usernameStatus": "ACTIVE", "verified": true } ], "username": "loginId", "usernameStatus": "ACTIVE", "verified": true } } ``` ### 2.2 Add secret/credentials API This API will add secret/credentials in the vault. Below are the available parameters of api. * **type:** type of secret/credential **(Eg.** WhatsappGupshup/Headers) * **secretBody:** key value pair of credentials * credentials for type **WhatsappGupshup** should be as follows: ``` "secretBody": { "usernameHSM": "usernameHSMValue", "passwordHSM": "passwordHSMValue", "username2Way": "username2WayValue", "password2Way": "password2WayValue" } ``` * credentials for type **Headers** should be as follows:<br> ``` "secretBody": { "key1": "value1", "key2": "value2" } ``` * **variableName:** name of variable against which this data is being saved, it will be used in [add adapter api](https://uci.sunbird.org/use/developer/api-documentation/pages/iyh1Cgn2i7ue92G4ym2v#2.1-add-adapter) configuration. it is used to fetch secretBody data from vault when added in adapter. Use below curl to save credentials in vault. ``` curl --location -g --request POST 'http://143.110.183.73:3001/admin/secret' \ --header 'ownerId: 8f7ee860-0163-4229-9d2a-01cef53145ba' \ --header 'ownerOrgId: org1' \ --header 'Authorization: Bearer loginToken' \ --header 'asset: secret' \ --header 'Content-Type: application/json' \ --data-raw '{ "secretBody": { "serviceKey": "secretKeyValue" }, "type": "Headers", "variableName": "uci-firebase-notification" }' ``` # Database Schema * [**Cassandra Xmessage Database**](/use/developer/database-schema/database-cassandra) * [**Postgres Forms Database**](/use/developer/database-schema/database-formsdb) * [**Postgres User Analytics Database**](/use/developer/database-schema/database-user-analytics) * [**Postgres Comms (UCI-API) Database**](/use/developer/database-schema/database-comms) # Cassandra Xmessage Database ### 1. Introduction At UCI, we leverage Cassandra as our database solution due to its distributed nature and remarkable data access speed. Cassandra's ability to distribute data across nodes enables seamless parallel data access, enhancing both read and write performance. we maintain a comprehensive history of bot conversations using the following schema. ### 2. Table Schema * Table Name - XMessage | Column Name | Type | Description | | ------------ | ------------- | ------------------------------------------------------------------------- | | id | UUID | Unique identifier for the record. | | userId | Text | User identifier associated with the message. | | fromId | Text | Identifier for the sender of the message. | | channel | Text | Channel through which the message was sent. | | provider | Text | Provider of the messaging service. | | timestamp | LocalDateTime | Date and time when the message was timestamped. | | messageState | Text | Current state or status of the message. | | xMessage | Text | Content of the message. | | app | Text | App is the name of bot | | auxData | Text | Currently not using | | messageId | Text | Unique identifier for fcm callback. | | replyId | Text | We can store id which provied by your provider like gupshup, netcore etc. | | causeId | Text | Same as the messageId | | sessionId | UUID | Random UUID | | ownerOrgId | Text | Identifier for the owning organization. | | ownerId | Text | Identifier for the owner of the message. | | botUuid | UUID | Unique identifier for the associated bot. | | tags | List < Text > | List of tags or labels associated with the message. | | respMsgId | Text | Identifier for a response message (FCM uses). | | remarks | Text | Additional remarks or comments about the message. | | --- | | | # Postgres Forms Database ### Introduction This database is built on PostgreSQL and manages all ODK questions and answers, as well as their respective states. It tracks the questions posed to users and the anticipated follow-up inquiries. ### 1. Table Schema * Table Name - question | Column Name | Type | Description | | -------------- | --------- | ------------------------------------------------------------------------ | | id | UUID | Unique identifier for the record. | | form\_id | Varchar | This is odk formid, while we create odk form we added in the xml. | | form\_version | Varchar | This is odk form version, while we create odk form we added in the xml. | | x\_path | Varchar | This is path of question get from odk form. | | question\_type | Varchar | Type of the question like String, Single Select and Multi-Select. | | meta | Jsonb | JSON data structure its contain the question which we sent to our users. | | updated | Timestamp | Timestamp indicating when the record was last updated. | | created | Timestamp | Timestamp indicating when the record was created. | | --- | | | ### 2. Table Schema * Table Name - xmessage | Column Name | Type | Description | | ----------------- | ------- | ------------------------------------------------------------------------------- | | id | Long | Unique identifier for the record. | | phone\_no | Varchar | Phone number associated with the user. | | message | Varchar | The content of the message which we sent to user. | | is\_last\_message | Boolean | A boolean flag indicating if this is the last response in a series of messages. | | --- | | | ### 3. Table Schema * Table Name - xmessage\_state | Column Name | Type | Description | | --------------- | ------- | ----------------------------------------------------------- | | id | Long | Unique identifier for the record. | | phoneNo | Varchar | Phone number associated with the user. | | state | Text | XML data for previous question. | | previous\_path | Varchar | Path or location of the previous question. | | bot\_form\_name | Varchar | Name or identifier of the bot form associated with the bot. | | --- | | | # Postgres User Analytics Database ### Introduction This database is built using PostgreSQL, and its primary role is to keep track of the delivery status of notifications and their current status. It helps us generate final reports that show which notifications were sent, which users received them, and which users clicked on them. We use this database to store information about messages that were successfully delivered and messages that were read by users. ### 1. Table Schema * Table Name - delivery\_report | Column Name | Type | Description | | -------------- | --------- | ------------------------------------------------------------------------------------------------------------------------------------------------------- | | id | UUID | Unique identifier for the record. | | external\_id | Varchar | A unique identifier generated by the transaction layer when sending notifications. This identifier is used on the Android side to manage notifications. | | user\_id | Varchar | Identifier of the user its kind of Mobile/Phone number. | | bot\_id | Varchar | This is the bot id to identifiy bot related information. | | bot\_name | Varchar | This is the bot name to show the bot name on the ui. | | fcm\_token | Varchar | Firebase Cloud Messaging (FCM) token for notifications. | | message\_state | Varchar | State or status of the message like DELIVERED, READ. | | cass\_id | Varchar | Cassandra id to check the notification related information like xmessage, fcm messageid etc. | | created\_on | Timestamp | Timestamp indicating when the record was created. | | --- | | | # Postgres Comms Database ### Introduction In this database, we utilize PostgreSQL to store all the pertinent information related to our chatbot. This includes details such as the adapter we are using, the type of transformer employed, and the number of users to whom we will send notifications or SMS messages. In essence, this database exclusively contains information about the chatbot. ### 1. Table Schema * Table Name - adapter | Column Name | Type | Description | | ----------- | --------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | id | UUID | Unique identifier for the record. | | createdAt | Timestamp | Timestamp indicating when the record was created. | | updatedAt | Timestamp | Timestamp indicating when the record was last updated. | | channel | Text | This field identifies the communication channel utilized, such as SMS, WhatsApp, or web-based platforms, among others. | | provider | Text | Name of the service provider, such as Netcore, Gupshup, Firebase, etc., responsible for delivering messages or services. | | config | Jsonb | JSON data structure storing configuration information related to service providers. This field may include vault key credentials and other provider-specific settings. | | name | Text | This field store name of adapter | | --- | | | ### 2. Table Schema * Table Name - Service | Column Name | Type | Description | | ----------- | --------- | ------------------------------------------------------------------------------------------------------------------------------- | | id | UUID | Unique identifier for the service record. | | createdAt | Timestamp | Timestamp indicating when the service record was created. | | updatedAt | Timestamp | Timestamp indicating when the service record was last updated. | | type | Text | Type of the service like odk, gql, get. use for user segment. | | config | JsonB | JSON data structure storing configuration information related to the service like user segment url, cadence, total records etc. | | name | Text | Name or title of the service. | | --- | | | ### 3. Table Schema * Table Name - Transformer | Column Name | Type | Description | | ----------- | --------- | --------------------------------------------------------------------------------- | | id | UUID | Unique identifier for the transformer record. | | createdAt | Timestamp | Timestamp indicating when the transformer record was created. | | updatedAt | Timestamp | Timestamp indicating when the transformer record was last updated. | | name | Text | Name or title associated with the transformer. | | tags | Text | Tags or labels associated with the transformer like Odk, Generic, Broadcast. | | config | Jsonb | JSON data structure storing configuration information related to the transformer. | | serviceId | UUID | Foreign key referencing the service to which the transformer is related. | | --- | | | ### 4. Table Schema * Table Name - TransformerConfig | Column Name | Type | Description | | ----------------- | --------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------- | | id | UUID | Unique identifier for the transformer configuration record. | | createdAt | Timestamp | Timestamp indicating when the configuration record was created. | | updatedAt | Timestamp | Timestamp indicating when the configuration record was last updated. | | transformerId | UUID | Foreign key referencing the associated transformer. | | meta | Jsonb | JSON data structure storing additional metadata related to the configuration like title, body, hiddenFields, formID, form, serviceClass, templateType etc. | | conversationLogic | UUID | Foreign key referencing the conversation logic associated with the configuration. | | --- | | | ### 5. Table Schema * Table Name - ConversationLogic | Column Name | Type | Description | | ----------- | --------- | ------------------------------------------------------------------------- | | id | UUID | Unique identifier for the conversation logic record. | | createdAt | Timestamp | Timestamp indicating when the conversation logic record was created. | | updatedAt | Timestamp | Timestamp indicating when the conversation logic record was last updated. | | description | Text | Description providing details about the conversation logic. | | adapterid | UUID | Foreign key referencing the associated adapter. | | name | Text | Name or title associated with the conversation logic. | | --- | | | ### 6. Table Schema * Table Name - bot | Column Name | Type | Description | | --------------- | --------- | ------------------------------------------------------------------------------------------------ | | id | UUID | Unique identifier for the bot record. | | createdAt | Timestamp | Timestamp indicating when the bot record was created. | | updatedAt | Timestamp | Timestamp indicating when the bot record was last updated. | | name | Text | The name of the bot. | | startingMessage | Text | The initial message or greeting used by the bot when interacting with users. | | ownerID | Text | Identifier of the owner or creator of the bot. | | ownerOrgID | Text | Identifier of the organization to which the bot owner belongs. | | purpose | Text | A brief description of the purpose or function of the bot. | | description | Text | A detailed description of the bot, including its capabilities and use cases. | | startDate | Date | The date when the bot became active or started operating. | | endDate | Date | The date when the bot's operation is scheduled to end or expire. | | status | BotStatus | The current status of the bot, indicating whether it is enabled or disabled. (ENABLED, DISABLED) | | tags | Text | Tags or labels associated with the bot for categorization or organization. | | botImage | Text | Bot image icon. | | --- | | | ### 7. Table Schema * Table Name - UserSegment | Column Name | Type | Description | | ---------------- | --------- | --------------------------------------------------------------------------------------- | | id | UUID | Unique identifier for the user segment record. | | createdAt | Timestamp | Timestamp indicating when the user segment record was created. | | updatedAt | Timestamp | Timestamp indicating when the user segment record was last updated. | | name | Text | Name or title associated with the user segment. | | description | Text | Description providing details about the user segment. | | count | Int | Count or number of users in the segment. | | category | Text | Category or type of the user segment. | | allServiceID | UUID | Unique identifier referencing the associated service for all users in the segment. | | byPhoneServiceID | UUID | Unique identifier referencing the associated service for users in the segment by phone. | | byIDServiceID | UUID | Unique identifier referencing the associated service for users in the segment by ID. | | botId | UUID | Unique identifier referencing the associated bot for the user segment. | | --- | | | # Contribution Guide [**Your first PR**](/use/developer/contribution-guide/your-first-pr) [**Contribute an Adapter**](/use/developer/contribution-guide/create-an-adapter) [**Adapter Implementation**](/use/developer/contribution-guide/adapter-implementation) [**Create a Transformer**](/use/developer/contribution-guide/create-a-transformer) # Your first PR Pull requests with patches, improvements, new features are a great help. Follow the steps to get your work included in the project. #### 1. Fork the project, clone your fork, and add the UCI-PWA remote: ``` # Clone your fork of the repo into the current directory git clone https://github.com/<your-username>/inbound.git # Navigate to the cloned directory cd inbound # Assign the original repo to a remote called "upstream" git remote add upstream https://github.com/samagra-comms/inbound.git ``` #### 2. Get the latest changes from upstream: ``` git checkout master git pull upstream master ``` #### 3. Create a new branch from the master/latest working branch to contain your changes. Best way is to call is to follow the type described in Git Commit Conventions stated above: /\<description/scope/topic> `git checkout -b <topic-branch-name>` Example: `git checkout -b mike/buckets-undefined-index` Or `git checkout -b mike/fix` #### 4. Commit your changes Commit your changes in logical chunks. Please adhere to **Git Commit Conventions** and **Coding guidelines** or your code is unlikely to be merged into the main project. Use Git's [interactive rebase](https://www.atlassian.com/git/tutorials/rewriting-history/git-rebase) feature to tidy up your commits before making them public. #### 5. Locally rebase the upstream master branch into your topic branch: `git pull --rebase upstream master` #### 6. Push your topic branch up to your fork: `git push origin <topic-branch-name>` #### 7. [Open a Pull Request](https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/proposing-changes-to-your-work-with-pull-requests/about-pull-requests) with a clear title and description against the **master/latest** branch. # Contribute an Adapter ## 1. Overview Adapters convert information provided by channels (SMS, Whatsapp) for each specific provider to xMessages and vise versa. ## 2. How to setup adapter Setup the adapter repository to start working on it. Follow the steps given to add a new adapter & test it. 1. Fork the below repository & clone it. `https://github.com/samagra-comms/adapter/` 2. Checkout to the latest stable master branch. 3. Open the project in any IDE spring/intellij. 4. Follow [point 3](#3-creating-your-own-adapters) to create a new adapter. 5. Write test cases for inbound method `convertMessageToXMsg` & outbound method `processOutBoundMessageF`. 6. A simple example to test the send text message is given in the [file](https://github.com/samagra-comms/adapter/blob/master/src/test/java/com/uci/adapter/netcore/whatsapp/NetcoreWhatsappAdapterTest.java). ## 3. Creating your own Adapters All adapters are named as `<ProviderName><ChannelName>Adapter`; for example GupshupWhatsappAdapter. Adapters should extend `AbstractProvider` and implement `IProvider`. Thus, it needs to implement the following methods: * `public Mono<XMessage> convertMessageToXMsg(Object msg) // Converts API response object to XMessage` * `public Mono<XMessage> processOutBoundMessageF(XMessage nextMsg) //Converts XMessage object to API response and call it.` These methods are called by `inbound` and `outbound` services internally to process the incoming and outgoing messages. All adapters with the above implementation will be valid. An example adapter can be found [here](https://github.com/samagra-comms/adapter/blob/master/src/main/java/com/uci/adapter/netcore/whatsapp/NetcoreWhatsappAdapter.java). ### 3.1. Incoming content Currently we accepts text/media/location/quickReplyButton/list messages. It should be implemeted according to the network provider(Netcore/Gupshup) [documentation](https://wadocs.pepipost.com/webhooks/overview/incoming-message). To enable these, we have to add them in `convertMessageToXMsg` and convert these according to the [xMessage spec](/use/developer/uci-basics/xmessage-specification) ### 3.2. Outgoing Content We can send a text message to user. This is the basic usage of an adapter. When creating a new adapter we should start with this scenario. A sample code for the same is also added [here](#5.-sample-code). We can also send media/location/quickReplyButton/list messages to user. As we are using the ODK forms, we use bind::stylingTags, bind::caption to show the media file or to show the select choices as list/quick reply buttons. There are a few constraints which will be applied with the quickReplyButton/list content. Below are the few styling tags we have allowed for now. * image //To send image file * audio //To send audio file * video //To send video file * document //To send document file * buttonsForListItems //To send choices as quick reply buttons * list //To send choices as list To send this type of content to user, adapter should implement it according to the network provider(Netcore/Gupshup) [documentation](https://wadocs.pepipost.com/webhooks/overview/incoming-message). ## 4. Adapter Credentials To make a bot use this adapter we will have to later add this adapter configuration in databse using the [create adapter](https://uci.sunbird.org/use/developer/contribution-guide/pages/iyh1Cgn2i7ue92G4ym2v#2.1-create-adapter) API. This adapter may require you to have some credentials, which can be stored in a vault service. [Click here](/use/developer/api-documentation/vault-apis) to see how to add a secret in vault. ## 5. Sample Code Sample code to receive a text message from channel & reply to channel. 1. Create a new provider class which implements IProvider. * Implement **convertMessageToXMsg** method for converting incoming message from channel to XMessage format. * Implement **processOutBoundMessageF** method to reply to channel by converting XMessage format to channel message format. ```java public class ProviderChannelAdapter extends AbstractProvider implements IProvider { @Override public Mono<XMessage> convertMessageToXMsg(Object message) throws JAXBException, JsonProcessingException { WebMessage webMessage = (WebMessage) message; SenderReceiverInfo from = SenderReceiverInfo.builder().deviceType(DeviceType.PHONE).build(); SenderReceiverInfo to = SenderReceiverInfo.builder().userID("admin").build(); XMessage.MessageState messageState = XMessage.MessageState.REPLIED; MessageId messageIdentifier = MessageId.builder().build(); XMessagePayload xmsgPayload = XMessagePayload.builder().build(); xmsgPayload.setText(webMessage.getText()); XMessage.MessageType messageType= XMessage.MessageType.TEXT; from.setUserID(webMessage.getFrom()); /* To use later in outbound reply message's message id & to */ messageIdentifier.setChannelMessageId(webMessage.getMessageId()); messageIdentifier.setReplyId(webMessage.getFrom()); XMessage x = XMessage.builder() .to(to) .from(from) .channelURI("web") .providerURI("provider") .messageState(messageState) .messageId(messageIdentifier) .messageType(messageType) .timestamp(Timestamp.valueOf(LocalDateTime.now()).getTime()) .payload(xmsgPayload).build(); return Mono.just(x); } @Override public Mono<XMessage> processOutBoundMessageF(XMessage xMsg) throws Exception { String phoneNo = "91" +xMsg.getTo().getUserID(); OutboundMessage message = OutboundMessage.builder() .message(Message.builder() .title(xMsg.getPayload().getText()) .choices(xMsg.getPayload().getButtonChoices()) .msg_type("TEXT") .build()) .to(phoneNo) .messageId(xMsg.getMessageId().getChannelMessageId()) .build(); String url = "http://example.com/replyMessageToChannel"; return WebService.getInstance(). sendOutboundMessage(url, outboundMessage) .map(new Function<WebResponse, XMessage>() { @Override public XMessage apply(WebResponse webResponse) { if(webResponse != null){ xMsg.setMessageId(MessageId.builder().channelMessageId(webResponse.getId()).build()); xMsg.setMessageState(XMessage.MessageState.SENT); } return xMsg; } }); } } ``` 2. Create a web service class that handles api which is used to reply to channel. ```java @Service public class WebService { private final WebClient webClient; private static WebService webService = null; public WebService(){ this.webClient = WebClient.builder().build(); } public static WebService getInstance() { if (webService == null) { return new WebService(); } else { return webService; } } public Mono<PwaWebResponse> sendOutboundMessage(String url, OutboundMessage outboundMessage) { return webClient.post() .uri(url) .body(Mono.just(outboundMessage), OutboundMessage.class) .retrieve() .bodyToMono(WebResponse.class) .map(new Function<WebResponse, WebResponse>() { @Override public WebResponse apply(WebResponse webResponse) { if (webResponse != null) { System.out.println("MESSAGE RESPONSE " + webResponse.getMessage()); System.out.println("STATUS RESPONSE " + webResponse.getStatus()); System.out.println("MESSAGE ID RESPONSE " + webResponse.getId()); return webResponse; } else { return null; } } }).doOnError(new Consumer<Throwable>() { @Override public void accept(Throwable throwable) { System.out.println("ERROR IS " + throwable.getLocalizedMessage()); } }); } } ``` 3. Create a class for format which is accepted by reply to channel api Eg. **<http://example.com/replyMessageToChannel>**. ```java @Getter @Setter @Builder class OutboundMessage { private Message message; private String to; private String messageId; } ``` 4. Create a message format class for OutboundMessage class property. ```java @Getter @Setter @Builder class Message { String title; private ArrayList<ButtonChoice> choices; private String msg_type; } ``` 5. Create a class that accepts response from channel reply api. ```java @Getter @Setter class WebResponse { private String id; private String status; private String message; } ``` 6. Create a class that accepts format for message received from channel. ```java @Getter @Setter class WebMessage extends CommonMessage { String messageId; String text; String from; } ``` 7. If we want to fetch the credentials of the adapter from vault service, see the code of [FirebaseNotificationAdapter](https://github.com/samagra-comms/adapter/blob/master/src/main/java/com/uci/adapter/firebase/web/FirebaseNotificationAdapter.java) to check how to do this. ## 6. Adapter implementation After the adapter is created we should be able to implement it in inbound & outbound service for receiving and sending messages to user. [Click here](/use/developer/contribution-guide/adapter-implementation) to see the process & sample code on how to to do this. ## 7. List of Existing Adapter Implementations * [Gupshup-Whatsapp](https://github.com/samagra-comms/adapter/blob/master/src/main/java/com/uci/adapter/gs/whatsapp/GupShupWhatsappAdapter.java) * [Netcore-Whatsapp](https://github.com/samagra-comms/adapter/blob/master/src/main/java/com/uci/adapter/netcore/whatsapp/NetcoreWhatsappAdapter.java) * [UCI Web Channel](https://github.com/samagra-comms/adapter/blob/master/src/main/java/com/uci/adapter/pwa/PwaWebPortalAdapter.java) * [Firebase Adapter](https://github.com/samagra-comms/adapter/blob/master/src/main/java/com/uci/adapter/firebase/web/FirebaseNotificationAdapter.java) # Adapter Implementation ## 1. Overview A new adapter can be created using this [link](/use/developer/contribution-guide/create-an-adapter). Once the adapter is created we can incorporate this apater in inbound and outbound service for receiving and sending message to user respectively. Below is the process on how to implement a adapter in inbound & outbound. ## 2. Inbound Implementation Inbound receives the messages from a channel, and uses the channel adapter to convert it to XMessage format. Once the messages is converted it will get processed. When a new adapter is created we will have to add a class that will receive the messages from channel and use the adapter to convert this message to XMessage. In the next section of the document, we will see the sample code for the same. ### 2.1 Setup inbound Follow the steps below to setup the inbound repository to start working on it. 1. Fork the below repository & clone it. `https://github.com/samagra-comms/inbound/` 2. Checkout to the recent stable `master` branch. 3. Open the project in any IDE spring/intellij. 4. Follow the next steps. ### 2.2 Sample Code Create a class for the channel & provider for the adapter created. **Eg**. NetcoreWhatsappController. ``` @Slf4j @RestController @RequestMapping(value = "/provider") public class ProviderChannelController { @Value("${inboundProcessed}") private String inboundProcessed; @Value("${inbound-error}") private String inboundError; private ProviderChannelAdapter providerChannelAdapter; @Autowired public SimpleProducer kafkaProducer; @Autowired public XMessageRepository xmsgRepo; @Autowired public BotService botService; @Autowired public RedisCacheService redisCacheService; @Value("${outbound}") public String outboundTopic; @Value("${messageReport}") public String topicReport; @RequestMapping(value = "/channel", method = RequestMethod.POST, consumes = MediaType.APPLICATION_JSON_VALUE) public void providerChannel(@RequestBody WebMessage message) throws JsonProcessingException, JAXBException { providerChannelAdapter = ProviderChannelAdapter.builder().build(); XMsgProcessingUtil.builder() .adapter(providerChannelAdapter) .xMsgRepo(xmsgRepo) .inboundMessage(message) .topicFailure(inboundError) .topicSuccess(inboundProcessed) .kafkaProducer(kafkaProducer) .botService(botService) .redisCacheService(redisCacheService) .topicOutbound(outboundTopic) .topicReport(topicReport) .build() .process(); } } ``` Once the class is create, we can use the link **[http://host:port/provider/channel](https://uci.sunbird.org/use/developer/contribution-guide/http:/host:port/provider/channel)** to receive messages from user. A user will receive a message in below format. ``` { "messageId": "67630c26-e29e-46ae-acb6-756704362d84", "text": "Hi", "from": "98********" } ``` A sample api to receive messages from user is given below. ``` curl --location --request POST 'localhost:8085/provider/channel' \ --header 'Content-Type: application/json' \ --data-raw '{ "messageId": "67630c26-e29e-46ae-acb6-756704362d84", "text": "Hi", "from": "98********" }' ``` ### 2.3 Existing Implementations Below are some of the existing implementation code of currently available adapters. * [Netcore Whatspp](https://github.com/samagra-comms/inbound/blob/release-4.9.0/src/main/java/com/uci/inbound/netcore/NetcoreWhatsappConverter.java) * [Gupshup Whatsapp](https://github.com/samagra-comms/inbound/blob/release-4.9.0/src/main/java/com/uci/inbound/incoming/GupShupWhatsappConverter.java) * [UCI Web Channel](https://github.com/samagra-comms/inbound/blob/release-4.9.0/src/main/java/com/uci/inbound/pwa/web/PwaWebController.java) ## 3. Outbound Implementation Outbound converts the xMessage to the one that will be sent to the channel(sms/whatsapp). It will then be sent to the network provider who will send it to the channel using the implementation done in adapter. Outbound uses this [OutboundKafkaController](https://github.com/samagra-comms/outbound/blob/release-4.9.0/src/main/java/com/uci/outbound/consumers/OutboundKafkaController.java) class to send messages to user. There is class called [ProviderFactory](https://github.com/samagra-comms/adapter/blob/release-4.9.0/src/main/java/com/uci/adapter/provider/factory/ProviderFactory.java) which uses the channel & provider set in the XMessage and return the adapter object which will be used to call the **processOutBoundMessageF** method. ### 3.1 Sample code Add code to return adapter object based on the channel & provider set in XMessage. ``` if(provider.toLowerCase().equals("specifiedProvider") && channel.toLowerCase().equals("specifiedChannel")){ return ProviderChannelAdapter.builder().build(); } ``` Once the code is added, we should be able to use the new adapter to sent messages to user. # Create a Transformer ## 1. Overview Transformers transforms the previous xMessage from the user to one that needs to be sent next. It is essentially a microservice that based on the previous user action, returns a new xMessage that will then be shown to the user. This also enables conversion from one type All inbound messages pass through a transformer. If a transformer is not assigned, then a null transformer is assigned to the xMessage. Since the current implementation of MessageRosa is only in Java, currently there is a limitation on the number of languages you can build the transformer on which right now is just Java. Simply put a transformer looks something like this ![](https://1242145269-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FnC2042spZ2TIbqdRA37S%2Fuploads%2FE0tCPx9MtdBmC6L34MNc%2Ftransformer.jpg?alt=media\&token=5fc693ba-9c6a-45db-b507-d8015b64d560) ### 1.1 Responsibilities * Acts as a state machine and converts messages from one state to another. * Used to transform messages * Applies conversational logic (constraints or input format) ## 2. Creating your own Transformer All transformers are named as `<Action>Transformer`; for example PDFTransformer will generate a PDF from the previous XMessage and convert it to an XMessage. Adapters should extend `TransformerProvider`. Thus, it needs to implement the following methods: * ```java public XMessage transform(XMessage nextMsg) //Takes in XMessage and return the next message. ``` ## 3. Sample Code Sample code to transform text to lowercase characters before sending to outbound ```java public class DemoTransformer extends TransformerProvider { private final Flux<ReceiverRecord<String, String>> reactiveKafkaReceiver; @Value("${processOutbound}") private String processOutboundTopic; @EventListener(ApplicationStartedEvent.class) public void onMessage() { reactiveKafkaReceiver .doOnNext(new Consumer<ReceiverRecord<String, String>>() { @Override public void accept(ReceiverRecord<String, String> stringMessage) { try { XMessage msg = XMessageParser.parse(new ByteArrayInputStream(stringMessage.value().getBytes())); transform(msg) .subscribe(new Consumer<XMessage>() { @Override public void accept(XMessage transformedMessage) { if (transformedMessage != null) { try { kafkaProducer.send(processOutboundTopic, transformedMessage.toXML()); } catch (JAXBException e) { e.printStackTrace(); } } } }); } catch (JAXBException e) { e.printStackTrace(); } catch (NullPointerException e) { log.error("An error occured : "+e.getMessage() + " at line no : "+ e.getStackTrace()[0].getLineNumber() +" in class : "+e.getStackTrace()[0].getClassName()); } catch (Exception e) { e.printStackTrace(); } } }) .doOnError(new Consumer<Throwable>() { @Override public void accept(Throwable e) { System.out.println(e.getMessage()); log.error("KafkaFlux exception", e); } }).subscribe(); } /* Transform text to lower case */ public XMessage transform(XMessage nextMsg) { Payload payload = nextMsg.getPayload(); String text = payload.getText(); payload.setText(text.toLowerCase()); nextMsg.setPayload(payload); return nextMsg; } } ``` All transformers with the implementation same as above exmaple will be valid. A detailed example transformer can be found [here](https://github.com/samagra-comms/transformer/blob/release-4.9.0/src/main/java/com/uci/transformer/odk/ODKConsumerReactive.java). ## 4. Properties of a transformer * Transformers could call external services and wait for resposes from them to process message. These transformers are called `call-only` transformer. An example of this transformer is the PDF service transformer which calls the PDF service to check the status of queue of previous message and responds with a response containing PDF for the user. * All transformers need to be registered. All unregistered transformers will not be acted upon. This is what essentially adds a topic to broker on which the messages are pushed. Also this requires some basic config of the max time a transformer could take to process the message. * Scaling of transformers is done horizontally but the broker needs to know the number so tha partitions can be reconfigured. * Since it's part of a state machine. If the transformer is stuck it needs communicate to the Orchestrator to that it can be escalate. # Contact the administrator Please write to the Maintainer - * Chakshu (<chakshu@samagragovernance.in>) - Engineer * Saket (<saket@samagragovernance.in>) - PM * Sukhpreet (<sukhpreet@samagragovernance.in>) - PM # Release V2 ## Overview This release note contains all the feature additions, optimizations as well as changes that have been incorporated into the UCI ecosystem since the V1 version of UCI. ## New Features * [Extend /health endpoints to make them async and detailed](https://github.com/samagra-comms/uci-apis/issues/54): Health endpoints now provide a more detailed result including all the dependent services. * [Making query URL paginated](https://github.com/samagra-comms/broadcast-transformer/issues/1): The notification process now fans out in chunks instead of one at a time. This enables the UCI system to perform better and faster. A bigger page size means faster performance but a more CPU and RAM intensive process. * [In bot details API, add icon image URL parameter](https://github.com/samagra-comms/uci-apis/issues/56): Conversation bots now support adding a display icon during the creation of bots. * [Media upload API](https://github.com/samagra-comms/uci-apis/issues/55): Admin panel now supports uploading media files while creating a bot directly from the admin panel. * [Add headers to query url being sent with broadcast conversation logic](https://github.com/samagra-comms/uci-admin/issues/15): Segment URL queries now pass an auth token to retrieve users. * [Add a checkbox to create broadcast bot](https://github.com/samagra-comms/uci-admin/issues/14): Admin panel now supports a checkbox to create a broadcast bot along with a conversation bot. * [Selected option highlighted](https://github.com/samagra-comms/uci-web-channel/issues/92): Web-Channel now highlights the previous options selected by users. * [\[C4GT\] Add Audio support to web channel](https://github.com/samagra-comms/uci-web-channel/issues/98): Web-Channel now supports audio messages. * [Add Bot Icon and Media caption to the chatbot](https://github.com/samagra-comms/uci-web-channel/issues/111): Web-Channel now supports adding a caption to media. * [Bulk Data Insertion in Fusion Auth](https://github.com/samagra-comms/uci-apis/issues/99): FusionAuth service now supports bulk insertion. * [Create a bot with status as disabled by default](https://github.com/samagra-comms/uci-apis/issues/121): Disabled bots now return `ServiceUnavailableException` when triggered. * [Return bot list in sorted order by name in allContextual endpoint](https://github.com/samagra-comms/uci-apis/issues/120): `allContextual` API now returns bots in sorted order. * [Return purpose and description in botDetails API](https://github.com/samagra-comms/uci-apis/issues/119): Update API now allows updation of fields such as `purpose`, `description`, `endDate`. * [Cassandra Server Configuration Pick from Docker-Compose](https://github.com/samagra-comms/inbound/issues/70): Docker Compose has been updated for Cassandra so that it picks up env from docker-compose file. * [Add New Column in Cassandra for Notification Failure Reason](https://github.com/samagra-comms/dao/issues/17): Cassandra `xmessage` table now contains additional column to store notification failure. * [Dockerize the web-channel to onboard to the one script deployment plan](https://github.com/samagra-comms/uci-web-channel/issues/123): UCI Web-Channel has now been dockerized. * [Unify specs across UCI APIs](https://github.com/samagra-comms/uci-apis/issues/58): `/admin/search/internal` endpoint has been removed and only `/admin/search` is supported. * [Add order by parameter in search API](https://github.com/samagra-comms/uci-apis/issues/174): Search API now also supports sorting by field. * [Integrate Edit Bot in Admin Panel](https://github.com/samagra-comms/uci-admin/issues/42): Admin Dashboard now supports editing a bot. * [Add loader to conversation-NL App](https://github.com/samagra-comms/uci-web-channel/issues/128): Web Channel now shows a loader until bot is loaded. * [Handling of expired bots](https://github.com/samagra-comms/uci-apis/issues/163): Expired bots now show an expired message in web channel. * [Broadcast Direct Message APIs](https://github.com/samagra-comms/outbound/issues/52): Direct message APIs can now send messages in broadcast mode. * [Add endpoint to get bot report data](https://github.com/samagra-comms/uci-apis/issues/206): Bot Service now supports endpoint to get bot report data for notification. * [Setup monitoring for bot health](https://github.com/samagra-comms/uci-apis/issues/211): Endpoint added for bot health. * [Provide endpoints for deletion of bots](https://github.com/samagra-comms/uci-apis/issues/223): UCI-API now supports an endpoint for the deletion of disabled bots. ## Bug Fixes * [Some tests are failing on stable branch (outbound)](https://github.com/samagra-comms/outbound/issues/28): Removed obsolete tests from outbound service. * [Some test cases failing on stable branch (Broadcast transformer)](https://github.com/samagra-comms/broadcast-transformer/issues/4): Removed obsolete tests from Broadcast service. * [Some tests are failing in stable branch (utils)](https://github.com/samagra-comms/utils/issues/42): Fixed failing tests in the utils package. * [Some test cases failing on stable branch (inbound)](https://github.com/samagra-comms/inbound/issues/43): Fixed failing tests in inbound service. * [Some tests are failing on stable branch (orchestrator)](https://github.com/samagra-comms/orchestrator/issues/35): Removed obsolete tests from Orchestrator service. * [Some tests are failing on stable branch (message-rosa)](https://github.com/samagra-comms/message-rosa/issues/18): Fixed failing tests on message-rosa package. * [Some tests are failing on stable branch (transformer)](https://github.com/samagra-comms/transformer/issues/66): Fixed failing tests on transformer * [Fix {FileCDN}/getSignedUrl endpoints to accept fileName](https://github.com/samagra-comms/inbound/issues/51): FileCDN endpoints have been fixed to accept filename as a query parameter instead of static querying. * [\[Bug\] - Getting Exception Related to Unmarshaller](https://github.com/samagra-comms/orchestrator/issues/42): Marshaller error while converting XMessages parallelly has been resolved. * [\[Bug\] - Getting Exception Related to Kafka](https://github.com/samagra-comms/orchestrator/issues/43): Null pointer exception on Kafka has been resolved. * [\[Bug\] Duplication of Kafka topics from Orchestrator](https://github.com/samagra-comms/orchestrator/issues/58): Duplicate message consumption from kafka in orchestrator service has been fixed. * [Prevent same bot same starting message to be created](https://github.com/samagra-comms/uci-apis/issues/166): UCI-API now prevents bots with same name or starting message as that of an existing bot to be created. * [Fix src/modules/service/service.service.spec.ts test](https://github.com/samagra-comms/uci-apis/issues/142): Service test cases fixed. * [Increase Minio Signed URL Expiry Time for NL-APP](https://github.com/samagra-comms/inbound/issues/82): Minio urls for media do not expire now. * [\[Fix Test\] src/modules/conversation-logic/conversation-logic.controller.spec.ts](https://github.com/samagra-comms/uci-apis/issues/134): Conversation Logic controller tests have been fixed. * [Fix src/app.controller.spec.ts test](https://github.com/samagra-comms/uci-apis/issues/145): App controller tests have been fixed. * [Invalidate cache on transaction layer on bot update](https://github.com/samagra-comms/uci-apis/issues/202): Upading a bot now invalidates cache on transaction layer. ## Performance Improvements/Enhancements * [Remove manual instance creation of NetcoreWhatsappAdapter](https://github.com/samagra-comms/inbound/issues/49): Removes manual instance creation of NetcoreWhatsappAdapter and uses dependency injection instead. * [Add retry to FusionAuth in updateUser](https://github.com/samagra-comms/uci-apis/issues/87): Added retry mechanism to FusionAuth to prevent the abrupt failing of requests. * [Notification Pipeline](https://github.com/samagra-comms/outbound/issues/35): The notification pipeline has been revamped to process more than 2L notifications without crashing services. * [\[Bug\] - Cassandra Related Issues in Inbound](https://github.com/samagra-comms/inbound/issues/55): Cassandra load issue was resolved by optimizing the queries. * [Add Cache in allContextual API](https://github.com/samagra-comms/uci-apis/issues/71): Adds caching to allContextual API to enhance response time. * [Add logging to UCI-APIS](https://github.com/samagra-comms/uci-apis/issues/69): Service level logs have been added to all the endpoints along with the total response time of functions. * [Reduce JVM memory usage](https://github.com/samagra-comms/docker-deploy/issues/71): Dockerfile of java services have been modified to cap total memory usage and to use Semeru JVM instead of openjdk-jvm. * [Improve error logging in UCI-APIs](https://github.com/samagra-comms/uci-apis/issues/112): UCI-API service now returns proper error messages on upload failure. * [Cache cassandra result in conversation history](https://github.com/samagra-comms/inbound/issues/75): User conversation history from Cassandra is now being cached to improve performance. * [Remove old implementation of getLatestXMessage](https://github.com/samagra-comms/inbound/issues/74): `getLatestXMessage` query has been optimized to perform better. * [vault dependency removal](https://github.com/samagra-comms/uci-apis/issues/85): Removed mittwald vault package dependency due to errors. * [\[Bug\] - Getting Error in Aggregate Postgres](https://github.com/samagra-comms/transformer/issues/84): Aggregate db postgres connection pool has been added to prevent `too many clients` error. * [Create a separate Kafka topic for notifications in each service](https://github.com/samagra-comms/inbound/issues/85): Notifications and Conversations now use separate topics for Kafka. * [Fix conversation flow in the setup by install.py](https://github.com/samagra-comms/docker-deploy/issues/87): UCI docker deploy now provides a single script setup which automatically configures a test bot and other dependencies. ## Dependencies * [NL backend](https://github.com/Mission-Prerna/Nipun-Lakshya-App): NL backend systems that are responsible for managing users. ## Documentation * [Document changes done to Outbound](https://github.com/samagra-comms/outbound/issues/22): Updated Firebase adapter docs to include a table explaining all the required fields in the request. * [Modify Admin Console Manual](https://github.com/samagra-comms/community/issues/52): The UCI documentation now also includes a manual for the admin panel. * [Added a table explaining all the different fields that are used while creating a new conversation logic](https://github.com/samagra-comms/community/pull/28): Documentation has been updated to explain all the relevant query fields in conversation creation API. * [Create flow diagram of message flow in UCI on Draw.io](https://github.com/samagra-comms/uci-apis/issues/150): Documentation now consists of Draw\.io diagrams of different flows. * [\[Docs\] Update documentation of bot history api to match v2](https://github.com/samagra-comms/community/issues/37): Documentation for bot history api has been updated for v2. * [\[Docs\] Update documentation of bot setup api to match v2](https://github.com/samagra-comms/community/issues/36): Bot setup api has been updated in documentation to match v2. * [\[Docs\] Update Setting up first conversation according to v2](https://github.com/samagra-comms/community/issues/34): Documentation updated for setting up first conversation. * [\[Doc\] - Documentation for Database Schema](https://github.com/samagra-comms/dao/issues/18): Documentation added for database schema. * [Update UCI Setup Docs](https://github.com/samagra-comms/docker-deploy/issues/91): Documentation has been updated with the latest setup guide which provides a more streamlined and hassle free installation process. ## Breaking Changes * Old Admin Console is not compatible with V2 version of UCI. * V2 version of UCI depends on NL backend and will not work without it. * Gupshup provider has been removed from the pipeline. ## Contributors * [geeky-abhishek](https://github.com/geeky-abhishek): Abhishek Kumar Pal * [choxx](https://github.com/choxx): Abhishek Mishra * [RyanWalker277](https://github.com/RyanWalker277): Anvansh Singh * [devAyushDubey](https://github.com/devAyushDubey): Ayush Dubey * [ChakshuGautam](https://github.com/ChakshuGautam): Chakshu Gautam * [chinmoy12c](https://github.com/chinmoy12c): Chinmoy Chakraborty * [karntrehan](https://github.com/karntrehan): Karan Trehan * [pankajjangid05](https://github.com/pankajjangid05): Pankaj Jangid * [tushar5526](https://github.com/tushar5526): Tushar Gupta * [1010varun](https://github.com/1010varun): Varun Agarwal * [yuvrajsab](https://github.com/yuvrajsab): Yuvraj Sablania ## Feedback We welcome your feedback! If you encounter any issues or have suggestions for improvements, please [submit an issue](https://github.com/samagra-comms/community/issues) on our issue tracker. Thank you for using UCI! # Software License MIT License Copyright (c) 2021-22 Samagra | Transforming Governance Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. # Acceptable Use Policy By using the services provided to you ("Services") by UCI, you hereby agree not to use the Services in a way that is contrary or inconsistent with the rules and standards published by UCI from time to time, including as set forth in this Acceptable Use Policy (this "Policy"), or otherwise in violation of applicable law. By way of example, you shall not: * probe, scan, or test the vulnerability of any system or network; * breach or otherwise circumvent any security or authentication measures; * access, tamper with, or use non-public areas or parts of the Services, or shared areas of the Services you haven't been invited to or otherwise granted access to by UCI; * interfere with or disrupt any user, host, or network, for example by sending a virus, overloading, flooding, spamming, or mail-bombing any part of the Services; * access, search, or create accounts for the Services by any means other than our publicly supported interfaces (for example, "scraping" or creating accounts in bulk); * send unsolicited communications, promotions or advertisements, or spam; * send altered, deceptive or false source-identifying information, including "spoofing" or "phishing"; * promote or advertise products or services other than your own without appropriate authorization; * abuse referrals or promotions to get more paid features than deserved; * circumvent the scope of your paid subscription, if applicable; * sell the Services unless specifically authorized to do so; * publish or share materials that are unlawfully pornographic or indecent, or that contain acts of violence; * advocate bigotry or hatred against any person or group of people based on their race, religion, ethnicity, sex, gender identity, sexual preference, disability, or impairment; * violate the privacy or infringe the rights of others; or * otherwise take any action that UCI deems to be inconsistent with acceptable business practices or other standards of behavior, in each case in as determined by UCI in its sole discretion. Any violation of this Policy will result in immediate suspension and/or termination of your access to the Services and, if applicable, provide UCI with the right to terminate any other agreement between you and UCI then-in effect. # Terms of Service These terms of service ("Terms") govern your use and access to our client software. Your agreement is with Samagra | Transforming Governance, a corporation organized under the laws of \<To be updated> and its affiliate entities ("UCI", "Sunbird UCI", "we", "us", or "our"). * If you are using our Services on behalf of a third party, you hereby agree to these Terms on such third party's behalf, and you hereby confirm that you have authority to bind such third party by such agreement. IF YOU DO NOT AGREE TO BE BOUND TO THESE TERMS AND THE OTHER POLICIES, AGREEMENTS AND DOCUMENTS REFERENCED HEREIN, YOU SHOULD NOT USE OUR SERVICES. #### Your Responsibilities You are solely responsible and liable for your conduct and Your Data, which must comply with our [Acceptable Use Policy](https://uci.sunbird.org/engage/acceptable-use-policy). Please do not collect, copy, upload, download or share Your Data with ODK unless you have the right and authority to do so; any such unauthorized sharing of Your Data with ODK will be considered a violation of these Terms and may result in the termination of Your Account and any Subscriptions for Services then in effect. We may review your conduct and Your Data for compliance with these Terms, but we have no obligation to do so. You may use our Services only as permitted by applicable law, including export control laws and regulations. Finally, by using our services, you represent to us that you and Your Delegates comply with the requirements of digital consent as per applicable law, as specified in the Privacy Policy. If you and Your Delegates do not meet these requirements, you may not use the Services. #### Software Some of our Services allow you to download client software programs and tools ("Software") which may update automatically. So long as you comply with these Terms, we give you a limited, nonexclusive, nontransferable, revocable license to use the Software solely to access the Services. To the extent any component of the Software may be offered under an open-source license, we will make the terms of that license available to you and the provisions of that license may expressly override some of these Terms. You hereby agree not to reverse engineer, decode or decompile the Services, attempt to do so, or assist anyone in doing so. #### Service Updates From time to time, UCI may, in its sole discretion, add features or functions or provide bug fixes, patches, updates and upgrades to our Services, including its components, which may be provided automatically and without advance notice. If you do not want to receive such upgrades automatically, you may not use the Services. We sometimes release products and features that we are still testing and evaluating. Those Services have been marked "beta," "pre-release", "preview," "early access," or "evaluation" (or with words or phrases with similar meanings) (collectively, "Beta Services"), and may not be as reliable as UCI's other services, so please keep that in mind. Any Beta Services are not subject to any warranties or service level/uptime/downtime commitments that UCI may provide to you from time to time in connection with the Services. #### Indemnification You will indemnify, defend, and hold harmless UCI from and against all liabilities, damages, and costs (including settlement costs and reasonable attorneys' fees) arising out of any claim by a third party against UCI and its Affiliates regarding: (a) Your Data; (b) your use of the Services. #### Services "AS IS" We strive to provide great Services, but there are certain things that we cannot guarantee. TO THE FULLEST EXTENT PERMITTED BY LAW, UCI AND ITS AFFILIATES, SUPPLIERS AND DISTRIBUTORS MAKE NO WARRANTIES ABOUT THE SERVICES. THE SERVICES ARE PROVIDED "AS IS". WE ALSO DISCLAIM ANY WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NON-INFRINGEMENT, WHETHER EXPRESS OR IMPLIED BY LAW, COURSE OF DEALING, COURSE OF PERFORMANCE, USAGE OF TRADE OR OTHERWISE. #### Limitation of Liability WE DO NOT EXCLUDE OR LIMIT OUR LIABILITY TO YOU WHERE IT WOULD BE ILLEGAL TO DO SO—THIS INCLUDES ANY LIABILITY FOR UCI'S OR ITS AFFILIATES' FRAUD OR GROSS NEGLIGENCE IN PROVIDING THE SERVICES. THIS PARAGRAPH DOES NOT AFFECT CONSUMER RIGHTS THAT CAN NOT BE WAIVED OR LIMITED BY ANY CONTRACT OR AGREEMENT. IN NO EVENT WILL UCI, ITS AFFILIATES, SUPPLIERS OR DISTRIBUTORS BE LIABLE TO YOU OR ANY THIRD PARTY FOR ANY ANY INDIRECT, SPECIAL, INCIDENTAL, PUNITIVE, EXEMPLARY OR CONSEQUENTIAL DAMAGES, OR ANY LOSS OF USE, DATA, BUSINESS, REVENUES, OR PROFITS (IN EACH CASE WHETHER DIRECT OR INDIRECT), REGARDLESS OF LEGAL THEORY, AND IN EACH CASE REGARDLESS OF WHETHER SUCH DAMAGES WERE FORESEEABLE AND WHETHER OR NOT UCI HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES, AND NOTWITHSTANDING THE FAILURE OF ANY AGREED OR OTHER REMEDY OF ITS ESSENTIAL PURPOSE. IF YOU USE THE SERVICES FOR ANY COMMERCIAL, BUSINESS OR RE-SALE PURPOSE, UCI, ITS AFFILIATES, SUPPLIERS OR DISTRIBUTORS WILL HAVE NO LIABILITY TO YOU FOR ANY LOSS OF PROFIT, LOSS OF BUSINESS, BUSINESS INTERRUPTION, OR LOSS OF BUSINESS OPPORTUNITY. UCI AND ITS AFFILIATES ARE NOT RESPONSIBLE FOR THE CONDUCT, WHETHER ONLINE OR OFFLINE, OF ANY USER OF THE SERVICES. #### Entire Agreement These Terms, together with the [Software Licence](/engage/software-license), the [Acceptable Use Policy](/engage/acceptable-use-policy), and the other documents and materials referenced herein or otherwise published by UCI from time to time governing use of the Services, constitute the entire agreement between you and UCI with respect to the subject matter of these Terms, and supersede and replace any other prior or contemporaneous agreements, understandings, or terms and conditions applicable to the subject matter of these Terms except as otherwise agreed in writing between you and UCI. These Terms create no third party beneficiary rights. #### Waiver, Severability & Assignment UCI's failure to enforce a provision is not a waiver of its right to do so at a later date or time. If any portion of these Terms is found unenforceable, the remaining provisions of the Terms will remain in full effect and an enforceable term will be substituted reflecting our intent as UCI as possible. You may not assign any of your rights under these Terms, and any such attempt will be void. UCI may assign its rights to any of its affiliates or subsidiaries, or to any successor in interest of any business associated with the Services. #### Modifications We may revise these Terms from time to time and the most current version will always be posted on the UCI's website. If a revision, in UCI's sole discretion, is material, we will notify you (by, for example, sending an email to the email address associated with your account). You are responsible for checking these postings regularly. By continuing to access or use the Services after revisions become effective, you agree to be bound by the revised Agreement. If you do not agree to the revised terms, you may not access or otherwise use the Services.