Model: SoLU Model: 6 Layers, 3072 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 2780 in Layer 3

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-6l')

Text #0

Max Range: 0.2558. Min Range: -0.2558

Max Act: 0.2558. Min Act: -0.0001

Data Index: 885031 (C4 (Web Text))

Max Activating Token Index: 80

Text #1

Max Range: 0.2558. Min Range: -0.2558

Max Act: 0.2404. Min Act: -0.0001

Data Index: 717634 (C4 (Web Text))

Max Activating Token Index: 13

Text #2

Max Range: 0.2558. Min Range: -0.2558

Max Act: 0.2387. Min Act: -0.0001

Data Index: 639495 (C4 (Web Text))

Max Activating Token Index: 673

Text #3

Max Range: 0.2558. Min Range: -0.2558

Max Act: 0.2426. Min Act: -0.0001

Data Index: 96295 (C4 (Web Text))

Max Activating Token Index: 524

Text #4

Max Range: 0.2558. Min Range: -0.2558

Max Act: 0.2356. Min Act: -0.0001

Data Index: 377623 (C4 (Web Text))

Max Activating Token Index: 732

Text #5

Max Range: 0.2558. Min Range: -0.2558

Max Act: 0.2494. Min Act: -0.0001

Data Index: 1343554 (C4 (Web Text))

Max Activating Token Index: 588

Text #6

Max Range: 0.2558. Min Range: -0.2558

Max Act: 0.2246. Min Act: -0.0001

Data Index: 768692 (C4 (Web Text))

Max Activating Token Index: 170

Text #7

Max Range: 0.2558. Min Range: -0.2558

Max Act: 0.2171. Min Act: -0.0001

Data Index: 773121 (C4 (Web Text))

Max Activating Token Index: 843

Text #8

Max Range: 0.2558. Min Range: -0.2558

Max Act: 0.2175. Min Act: -0.0001

Data Index: 610848 (C4 (Web Text))

Max Activating Token Index: 267

Text #9

Max Range: 0.2558. Min Range: -0.2558

Max Act: 0.2166. Min Act: -0.0001

Data Index: 1309292 (C4 (Web Text))

Max Activating Token Index: 608

Text #10

Max Range: 0.2558. Min Range: -0.2558

Max Act: 0.2196. Min Act: -0.0001

Data Index: 794934 (C4 (Web Text))

Max Activating Token Index: 491

Text #11

Max Range: 0.2558. Min Range: -0.2558

Max Act: 0.2049. Min Act: -0.0001

Data Index: 1355113 (C4 (Web Text))

Max Activating Token Index: 554

Text #12

Max Range: 0.2558. Min Range: -0.2558

Max Act: 0.2047. Min Act: -0.0001

Data Index: 1162534 (C4 (Web Text))

Max Activating Token Index: 904

Text #13

Max Range: 0.2558. Min Range: -0.2558

Max Act: 0.2170. Min Act: -0.0001

Data Index: 1196047 (C4 (Web Text))

Max Activating Token Index: 408

Text #14

Max Range: 0.2558. Min Range: -0.2558

Max Act: 0.2038. Min Act: -0.0001

Data Index: 510772 (C4 (Web Text))

Max Activating Token Index: 416

Text #15

Max Range: 0.2558. Min Range: -0.2558

Max Act: 0.2016. Min Act: -0.0001

Data Index: 928393 (C4 (Web Text))

Max Activating Token Index: 576

Text #16

Max Range: 0.2558. Min Range: -0.2558

Max Act: 0.2077. Min Act: -0.0001

Data Index: 310404 (C4 (Web Text))

Max Activating Token Index: 844

Text #17

Max Range: 0.2558. Min Range: -0.2558

Max Act: 0.1921. Min Act: -0.0001

Data Index: 401687 (C4 (Web Text))

Max Activating Token Index: 786

Text #18

Max Range: 0.2558. Min Range: -0.2558

Max Act: 0.1992. Min Act: -0.0001

Data Index: 206488 (C4 (Web Text))

Max Activating Token Index: 622

Text #19

Max Range: 0.2558. Min Range: -0.2558

Max Act: 0.1978. Min Act: -0.0001

Data Index: 294468 (C4 (Web Text))

Max Activating Token Index: 228