Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 271 in Layer 7

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 2.8250. Min Range: -2.8250

Max Act: 2.8250. Min Act: -0.0000

Data Index: 933806 (C4 (Web Text))

Max Activating Token Index: 38

Text #1

Max Range: 2.8250. Min Range: -2.8250

Max Act: 2.7706. Min Act: -0.0000

Data Index: 1167666 (C4 (Web Text))

Max Activating Token Index: 408

Text #2

Max Range: 2.8250. Min Range: -2.8250

Max Act: 2.5790. Min Act: -0.0000

Data Index: 1220785 (C4 (Web Text))

Max Activating Token Index: 454

Text #3

Max Range: 2.8250. Min Range: -2.8250

Max Act: 2.5011. Min Act: -0.0000

Data Index: 661923 (C4 (Web Text))

Max Activating Token Index: 229

Text #4

Max Range: 2.8250. Min Range: -2.8250

Max Act: 2.4849. Min Act: -0.0000

Data Index: 822974 (C4 (Web Text))

Max Activating Token Index: 875

Text #5

Max Range: 2.8250. Min Range: -2.8250

Max Act: 2.4707. Min Act: -0.0000

Data Index: 1264671 (C4 (Web Text))

Max Activating Token Index: 902

Text #6

Max Range: 2.8250. Min Range: -2.8250

Max Act: 2.4186. Min Act: -0.0000

Data Index: 520275 (C4 (Web Text))

Max Activating Token Index: 815

Text #7

Max Range: 2.8250. Min Range: -2.8250

Max Act: 2.3699. Min Act: -0.0000

Data Index: 711332 (C4 (Web Text))

Max Activating Token Index: 253

Text #8

Max Range: 2.8250. Min Range: -2.8250

Max Act: 2.3255. Min Act: -0.0000

Data Index: 1349294 (C4 (Web Text))

Max Activating Token Index: 485

Text #9

Max Range: 2.8250. Min Range: -2.8250

Max Act: 2.3186. Min Act: -0.0000

Data Index: 580779 (C4 (Web Text))

Max Activating Token Index: 1002

Text #10

Max Range: 2.8250. Min Range: -2.8250

Max Act: 2.2986. Min Act: -0.0000

Data Index: 489624 (C4 (Web Text))

Max Activating Token Index: 143

Text #11

Max Range: 2.8250. Min Range: -2.8250

Max Act: 2.2606. Min Act: -0.0000

Data Index: 459656 (C4 (Web Text))

Max Activating Token Index: 1021

Text #12

Max Range: 2.8250. Min Range: -2.8250

Max Act: 2.2750. Min Act: -0.0000

Data Index: 303623 (C4 (Web Text))

Max Activating Token Index: 16

Text #13

Max Range: 2.8250. Min Range: -2.8250

Max Act: 2.1362. Min Act: -0.0000

Data Index: 744885 (C4 (Web Text))

Max Activating Token Index: 297

Text #14

Max Range: 2.8250. Min Range: -2.8250

Max Act: 2.2079. Min Act: -0.0000

Data Index: 1130252 (C4 (Web Text))

Max Activating Token Index: 764

Text #15

Max Range: 2.8250. Min Range: -2.8250

Max Act: 2.2038. Min Act: -0.0000

Data Index: 969315 (C4 (Web Text))

Max Activating Token Index: 811

Text #16

Max Range: 2.8250. Min Range: -2.8250

Max Act: 2.3001. Min Act: -0.0000

Data Index: 184148 (C4 (Web Text))

Max Activating Token Index: 908

Text #17

Max Range: 2.8250. Min Range: -2.8250

Max Act: 2.3078. Min Act: -0.0000

Data Index: 1045566 (C4 (Web Text))

Max Activating Token Index: 584

Text #18

Max Range: 2.8250. Min Range: -2.8250

Max Act: 2.1562. Min Act: -0.0000

Data Index: 970103 (C4 (Web Text))

Max Activating Token Index: 35

Text #19

Max Range: 2.8250. Min Range: -2.8250

Max Act: 2.1411. Min Act: -0.0000

Data Index: 571274 (C4 (Web Text))

Max Activating Token Index: 523