Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1371 in Layer 3

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 1.0163. Min Range: -1.0163

Max Act: 1.0163. Min Act: -0.0001

Data Index: 1109146 (C4 (Web Text))

Max Activating Token Index: 746

Text #1

Max Range: 1.0163. Min Range: -1.0163

Max Act: 0.8953. Min Act: -0.0001

Data Index: 1175984 (C4 (Web Text))

Max Activating Token Index: 955

Text #2

Max Range: 1.0163. Min Range: -1.0163

Max Act: 0.9327. Min Act: -0.0001

Data Index: 989227 (C4 (Web Text))

Max Activating Token Index: 712

Text #3

Max Range: 1.0163. Min Range: -1.0163

Max Act: 0.8830. Min Act: -0.0001

Data Index: 1009475 (C4 (Web Text))

Max Activating Token Index: 685

Text #4

Max Range: 1.0163. Min Range: -1.0163

Max Act: 0.8086. Min Act: -0.0001

Data Index: 1315700 (C4 (Web Text))

Max Activating Token Index: 446

Text #5

Max Range: 1.0163. Min Range: -1.0163

Max Act: 0.8269. Min Act: -0.0001

Data Index: 843341 (C4 (Web Text))

Max Activating Token Index: 201

Text #6

Max Range: 1.0163. Min Range: -1.0163

Max Act: 0.8009. Min Act: -0.0001

Data Index: 703506 (C4 (Web Text))

Max Activating Token Index: 216

Text #7

Max Range: 1.0163. Min Range: -1.0163

Max Act: 0.8758. Min Act: -0.0001

Data Index: 902754 (C4 (Web Text))

Max Activating Token Index: 345

Text #8

Max Range: 1.0163. Min Range: -1.0163

Max Act: 0.7011. Min Act: -0.0001

Data Index: 411057 (C4 (Web Text))

Max Activating Token Index: 601

Text #9

Max Range: 1.0163. Min Range: -1.0163

Max Act: 0.8151. Min Act: -0.0001

Data Index: 105871 (C4 (Web Text))

Max Activating Token Index: 69

Text #10

Max Range: 1.0163. Min Range: -1.0163

Max Act: 0.7694. Min Act: -0.0001

Data Index: 734275 (C4 (Web Text))

Max Activating Token Index: 699

Text #11

Max Range: 1.0163. Min Range: -1.0163

Max Act: 0.8037. Min Act: -0.0001

Data Index: 1163194 (C4 (Web Text))

Max Activating Token Index: 980

Text #12

Max Range: 1.0163. Min Range: -1.0163

Max Act: 0.7668. Min Act: -0.0001

Data Index: 559963 (C4 (Web Text))

Max Activating Token Index: 742

Text #13

Max Range: 1.0163. Min Range: -1.0163

Max Act: 0.8520. Min Act: -0.0001

Data Index: 98844 (C4 (Web Text))

Max Activating Token Index: 122

Text #14

Max Range: 1.0163. Min Range: -1.0163

Max Act: 0.7460. Min Act: -0.0001

Data Index: 1169368 (C4 (Web Text))

Max Activating Token Index: 513

Text #15

Max Range: 1.0163. Min Range: -1.0163

Max Act: 0.7431. Min Act: -0.0001

Data Index: 871418 (C4 (Web Text))

Max Activating Token Index: 152

Text #16

Max Range: 1.0163. Min Range: -1.0163

Max Act: 0.7215. Min Act: -0.0001

Data Index: 1101150 (C4 (Web Text))

Max Activating Token Index: 220

Text #17

Max Range: 1.0163. Min Range: -1.0163

Max Act: 0.7609. Min Act: -0.0001

Data Index: 490738 (C4 (Web Text))

Max Activating Token Index: 730

Text #18

Max Range: 1.0163. Min Range: -1.0163

Max Act: 0.6959. Min Act: -0.0001

Data Index: 112449 (C4 (Web Text))

Max Activating Token Index: 263

Text #19

Max Range: 1.0163. Min Range: -1.0163

Max Act: 0.7776. Min Act: -0.0001

Data Index: 1186731 (C4 (Web Text))

Max Activating Token Index: 425