Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1179 in Layer 8

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 1.0043. Min Range: -1.0043

Max Act: 1.0043. Min Act: -0.0000

Data Index: 689647 (C4 (Web Text))

Max Activating Token Index: 718

Text #1

Max Range: 1.0043. Min Range: -1.0043

Max Act: 0.8873. Min Act: -0.0000

Data Index: 11655 (C4 (Web Text))

Max Activating Token Index: 924

Text #2

Max Range: 1.0043. Min Range: -1.0043

Max Act: 0.8105. Min Act: -0.0000

Data Index: 860826 (C4 (Web Text))

Max Activating Token Index: 378

Text #3

Max Range: 1.0043. Min Range: -1.0043

Max Act: 0.7589. Min Act: -0.0000

Data Index: 722756 (C4 (Web Text))

Max Activating Token Index: 421

Text #4

Max Range: 1.0043. Min Range: -1.0043

Max Act: 0.7427. Min Act: -0.0000

Data Index: 809039 (C4 (Web Text))

Max Activating Token Index: 684

Text #5

Max Range: 1.0043. Min Range: -1.0043

Max Act: 0.7179. Min Act: -0.0000

Data Index: 155491 (C4 (Web Text))

Max Activating Token Index: 657

Text #6

Max Range: 1.0043. Min Range: -1.0043

Max Act: 0.7454. Min Act: -0.0000

Data Index: 1212303 (C4 (Web Text))

Max Activating Token Index: 477

Text #7

Max Range: 1.0043. Min Range: -1.0043

Max Act: 0.6686. Min Act: -0.0000

Data Index: 1192513 (C4 (Web Text))

Max Activating Token Index: 136

Text #8

Max Range: 1.0043. Min Range: -1.0043

Max Act: 0.7093. Min Act: -0.0000

Data Index: 414829 (C4 (Web Text))

Max Activating Token Index: 938

Text #9

Max Range: 1.0043. Min Range: -1.0043

Max Act: 0.7500. Min Act: -0.0000

Data Index: 480272 (C4 (Web Text))

Max Activating Token Index: 708

Text #10

Max Range: 1.0043. Min Range: -1.0043

Max Act: 0.6861. Min Act: -0.0000

Data Index: 661429 (C4 (Web Text))

Max Activating Token Index: 346

Text #11

Max Range: 1.0043. Min Range: -1.0043

Max Act: 0.6526. Min Act: -0.0000

Data Index: 288749 (C4 (Web Text))

Max Activating Token Index: 895

Text #12

Max Range: 1.0043. Min Range: -1.0043

Max Act: 0.6512. Min Act: -0.0000

Data Index: 984620 (C4 (Web Text))

Max Activating Token Index: 342

Text #13

Max Range: 1.0043. Min Range: -1.0043

Max Act: 0.6593. Min Act: -0.0000

Data Index: 1136178 (C4 (Web Text))

Max Activating Token Index: 311

Text #14

Max Range: 1.0043. Min Range: -1.0043

Max Act: 0.6249. Min Act: -0.0000

Data Index: 1083954 (C4 (Web Text))

Max Activating Token Index: 511

Text #15

Max Range: 1.0043. Min Range: -1.0043

Max Act: 0.6142. Min Act: -0.0000

Data Index: 316123 (C4 (Web Text))

Max Activating Token Index: 453

Text #16

Max Range: 1.0043. Min Range: -1.0043

Max Act: 0.6176. Min Act: -0.0000

Data Index: 815066 (C4 (Web Text))

Max Activating Token Index: 853

Text #17

Max Range: 1.0043. Min Range: -1.0043

Max Act: 0.6097. Min Act: -0.0000

Data Index: 940343 (C4 (Web Text))

Max Activating Token Index: 574

Text #18

Max Range: 1.0043. Min Range: -1.0043

Max Act: 0.6047. Min Act: -0.0000

Data Index: 1058086 (C4 (Web Text))

Max Activating Token Index: 707

Text #19

Max Range: 1.0043. Min Range: -1.0043

Max Act: 0.6270. Min Act: -0.0000

Data Index: 633085 (C4 (Web Text))

Max Activating Token Index: 918