Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 3242 in Layer 4

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 0.1457. Min Range: -0.1457

Max Act: 0.1457. Min Act: -0.0001

Data Index: 431320 (C4 (Web Text))

Max Activating Token Index: 42

Text #1

Max Range: 0.1457. Min Range: -0.1457

Max Act: 0.1297. Min Act: -0.0001

Data Index: 1295668 (C4 (Web Text))

Max Activating Token Index: 53

Text #2

Max Range: 0.1457. Min Range: -0.1457

Max Act: 0.1280. Min Act: -0.0001

Data Index: 385004 (C4 (Web Text))

Max Activating Token Index: 456

Text #3

Max Range: 0.1457. Min Range: -0.1457

Max Act: 0.1228. Min Act: -0.0001

Data Index: 575546 (C4 (Web Text))

Max Activating Token Index: 525

Text #4

Max Range: 0.1457. Min Range: -0.1457

Max Act: 0.1213. Min Act: -0.0001

Data Index: 126051 (C4 (Web Text))

Max Activating Token Index: 934

Text #5

Max Range: 0.1457. Min Range: -0.1457

Max Act: 0.1163. Min Act: -0.0001

Data Index: 861751 (C4 (Web Text))

Max Activating Token Index: 608

Text #6

Max Range: 0.1457. Min Range: -0.1457

Max Act: 0.1079. Min Act: -0.0001

Data Index: 317524 (C4 (Web Text))

Max Activating Token Index: 324

Text #7

Max Range: 0.1457. Min Range: -0.1457

Max Act: 0.1093. Min Act: -0.0001

Data Index: 315152 (C4 (Web Text))

Max Activating Token Index: 643

Text #8

Max Range: 0.1457. Min Range: -0.1457

Max Act: 0.0927. Min Act: -0.0001

Data Index: 877851 (C4 (Web Text))

Max Activating Token Index: 340

Text #9

Max Range: 0.1457. Min Range: -0.1457

Max Act: 0.1060. Min Act: -0.0001

Data Index: 147892 (C4 (Web Text))

Max Activating Token Index: 40

Text #10

Max Range: 0.1457. Min Range: -0.1457

Max Act: 0.0796. Min Act: -0.0001

Data Index: 567499 (C4 (Web Text))

Max Activating Token Index: 925

Text #11

Max Range: 0.1457. Min Range: -0.1457

Max Act: 0.0930. Min Act: -0.0001

Data Index: 425963 (C4 (Web Text))

Max Activating Token Index: 691

Text #12

Max Range: 0.1457. Min Range: -0.1457

Max Act: 0.0944. Min Act: -0.0001

Data Index: 214051 (C4 (Web Text))

Max Activating Token Index: 795

Text #13

Max Range: 0.1457. Min Range: -0.1457

Max Act: 0.0891. Min Act: -0.0001

Data Index: 363404 (C4 (Web Text))

Max Activating Token Index: 178

Text #14

Max Range: 0.1457. Min Range: -0.1457

Max Act: 0.0880. Min Act: -0.0001

Data Index: 87193 (C4 (Web Text))

Max Activating Token Index: 327

Text #15

Max Range: 0.1457. Min Range: -0.1457

Max Act: 0.0860. Min Act: -0.0001

Data Index: 22592 (C4 (Web Text))

Max Activating Token Index: 897

Text #16

Max Range: 0.1457. Min Range: -0.1457

Max Act: 0.0859. Min Act: -0.0001

Data Index: 492026 (C4 (Web Text))

Max Activating Token Index: 333

Text #17

Max Range: 0.1457. Min Range: -0.1457

Max Act: 0.0802. Min Act: -0.0001

Data Index: 1000931 (C4 (Web Text))

Max Activating Token Index: 61

Text #18

Max Range: 0.1457. Min Range: -0.1457

Max Act: 0.0823. Min Act: -0.0001

Data Index: 897212 (C4 (Web Text))

Max Activating Token Index: 985

Text #19

Max Range: 0.1457. Min Range: -0.1457

Max Act: 0.0877. Min Act: -0.0001

Data Index: 11705 (C4 (Web Text))

Max Activating Token Index: 390