Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 711 in Layer 1

Load this data into an Interactive Neuroscope

See Documentation here

Hooked Transformer Loading: HookedTransformer.from_pretrained('solu-12l')

Click the toggle to see the full text!

Text #0

Max Range: 1.1261. Min Range: -1.1261

Max Act: 1.1261. Min Act: -0.0002

Data Index: 813218 (C4 (Web Text))

Max Activating Token Index: 883

Text #1

Max Range: 1.1261. Min Range: -1.1261

Max Act: 1.1212. Min Act: -0.0002

Data Index: 206289 (C4 (Web Text))

Max Activating Token Index: 3

Text #2

Max Range: 1.1261. Min Range: -1.1261

Max Act: 1.1212. Min Act: -0.0002

Data Index: 894692 (C4 (Web Text))

Max Activating Token Index: 3

Text #3

Max Range: 1.1261. Min Range: -1.1261

Max Act: 1.0855. Min Act: -0.0002

Data Index: 324729 (C4 (Web Text))

Max Activating Token Index: 14

Text #4

Max Range: 1.1261. Min Range: -1.1261

Max Act: 1.0909. Min Act: -0.0002

Data Index: 913902 (C4 (Web Text))

Max Activating Token Index: 415

Text #5

Max Range: 1.1261. Min Range: -1.1261

Max Act: 1.0618. Min Act: -0.0002

Data Index: 693554 (C4 (Web Text))

Max Activating Token Index: 1019

Text #6

Max Range: 1.1261. Min Range: -1.1261

Max Act: 1.0579. Min Act: -0.0002

Data Index: 853264 (C4 (Web Text))

Max Activating Token Index: 4

Text #7

Max Range: 1.1261. Min Range: -1.1261

Max Act: 1.0446. Min Act: -0.0002

Data Index: 680426 (C4 (Web Text))

Max Activating Token Index: 959

Text #8

Max Range: 1.1261. Min Range: -1.1261

Max Act: 1.0043. Min Act: -0.0002

Data Index: 191202 (C4 (Web Text))

Max Activating Token Index: 596

Text #9

Max Range: 1.1261. Min Range: -1.1261

Max Act: 1.0184. Min Act: -0.0002

Data Index: 195631 (C4 (Web Text))

Max Activating Token Index: 752

Text #10

Max Range: 1.1261. Min Range: -1.1261

Max Act: 1.0329. Min Act: -0.0002

Data Index: 834961 (C4 (Web Text))

Max Activating Token Index: 3

Text #11

Max Range: 1.1261. Min Range: -1.1261

Max Act: 0.9973. Min Act: -0.0002

Data Index: 43404 (C4 (Web Text))

Max Activating Token Index: 192

Text #12

Max Range: 1.1261. Min Range: -1.1261

Max Act: 0.9925. Min Act: -0.0002

Data Index: 136096 (C4 (Web Text))

Max Activating Token Index: 21

Text #13

Max Range: 1.1261. Min Range: -1.1261

Max Act: 1.0014. Min Act: -0.0002

Data Index: 551871 (C4 (Web Text))

Max Activating Token Index: 252

Text #14

Max Range: 1.1261. Min Range: -1.1261

Max Act: 0.9727. Min Act: -0.0002

Data Index: 828527 (C4 (Web Text))

Max Activating Token Index: 799

Text #15

Max Range: 1.1261. Min Range: -1.1261

Max Act: 0.9945. Min Act: -0.0002

Data Index: 863352 (C4 (Web Text))

Max Activating Token Index: 1008

Text #16

Max Range: 1.1261. Min Range: -1.1261

Max Act: 0.9990. Min Act: -0.0002

Data Index: 794465 (C4 (Web Text))

Max Activating Token Index: 466

Text #17

Max Range: 1.1261. Min Range: -1.1261

Max Act: 0.9945. Min Act: -0.0002

Data Index: 1356018 (C4 (Web Text))

Max Activating Token Index: 252

Text #18

Max Range: 1.1261. Min Range: -1.1261

Max Act: 0.9898. Min Act: -0.0002

Data Index: 1130945 (C4 (Web Text))

Max Activating Token Index: 710

Text #19

Max Range: 1.1261. Min Range: -1.1261

Max Act: 0.9916. Min Act: -0.0002

Data Index: 970028 (C4 (Web Text))

Max Activating Token Index: 479