Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 2212 in Layer 6

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 1.4048. Min Range: -1.4048

Max Act: 1.3989. Min Act: -0.0000

Data Index: 861249 (C4 (Web Text))

Max Activating Token Index: 494

Text #1

Max Range: 1.4048. Min Range: -1.4048

Max Act: 1.3044. Min Act: -0.0000

Data Index: 666970 (C4 (Web Text))

Max Activating Token Index: 792

Text #2

Max Range: 1.4048. Min Range: -1.4048

Max Act: 1.2975. Min Act: -0.0000

Data Index: 1182008 (C4 (Web Text))

Max Activating Token Index: 223

Text #3

Max Range: 1.4048. Min Range: -1.4048

Max Act: 1.4048. Min Act: -0.0000

Data Index: 1332682 (C4 (Web Text))

Max Activating Token Index: 175

Text #4

Max Range: 1.4048. Min Range: -1.4048

Max Act: 1.0738. Min Act: -0.0000

Data Index: 589136 (C4 (Web Text))

Max Activating Token Index: 544

Text #5

Max Range: 1.4048. Min Range: -1.4048

Max Act: 1.0627. Min Act: -0.0000

Data Index: 1359538 (C4 (Web Text))

Max Activating Token Index: 243

Text #6

Max Range: 1.4048. Min Range: -1.4048

Max Act: 0.9740. Min Act: -0.0000

Data Index: 119528 (C4 (Web Text))

Max Activating Token Index: 767

Text #7

Max Range: 1.4048. Min Range: -1.4048

Max Act: 0.9801. Min Act: -0.0000

Data Index: 827779 (C4 (Web Text))

Max Activating Token Index: 389

Text #8

Max Range: 1.4048. Min Range: -1.4048

Max Act: 1.0106. Min Act: -0.0000

Data Index: 441362 (C4 (Web Text))

Max Activating Token Index: 536

Text #9

Max Range: 1.4048. Min Range: -1.4048

Max Act: 0.9396. Min Act: -0.0000

Data Index: 714295 (C4 (Web Text))

Max Activating Token Index: 54

Text #10

Max Range: 1.4048. Min Range: -1.4048

Max Act: 1.0632. Min Act: -0.0000

Data Index: 84135 (C4 (Web Text))

Max Activating Token Index: 919

Text #11

Max Range: 1.4048. Min Range: -1.4048

Max Act: 1.0237. Min Act: -0.0000

Data Index: 226768 (C4 (Web Text))

Max Activating Token Index: 831

Text #12

Max Range: 1.4048. Min Range: -1.4048

Max Act: 0.9536. Min Act: -0.0000

Data Index: 616983 (C4 (Web Text))

Max Activating Token Index: 692

Text #13

Max Range: 1.4048. Min Range: -1.4048

Max Act: 0.9438. Min Act: -0.0000

Data Index: 1103762 (C4 (Web Text))

Max Activating Token Index: 273

Text #14

Max Range: 1.4048. Min Range: -1.4048

Max Act: 0.9148. Min Act: -0.0000

Data Index: 1209465 (C4 (Web Text))

Max Activating Token Index: 63

Text #15

Max Range: 1.4048. Min Range: -1.4048

Max Act: 0.9619. Min Act: -0.0000

Data Index: 602493 (C4 (Web Text))

Max Activating Token Index: 592

Text #16

Max Range: 1.4048. Min Range: -1.4048

Max Act: 0.9035. Min Act: -0.0000

Data Index: 1112932 (C4 (Web Text))

Max Activating Token Index: 249

Text #17

Max Range: 1.4048. Min Range: -1.4048

Max Act: 0.8583. Min Act: -0.0000

Data Index: 703708 (C4 (Web Text))

Max Activating Token Index: 270

Text #18

Max Range: 1.4048. Min Range: -1.4048

Max Act: 0.8938. Min Act: -0.0000

Data Index: 579740 (C4 (Web Text))

Max Activating Token Index: 415

Text #19

Max Range: 1.4048. Min Range: -1.4048

Max Act: 0.9164. Min Act: -0.0000

Data Index: 439250 (C4 (Web Text))

Max Activating Token Index: 108