Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 4378 in Layer 9

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.7148. Min Range: -0.7148

Max Act: 0.7148. Min Act: -0.0000

Data Index: 147936 (C4 (Web Text))

Max Activating Token Index: 838

Text #1

Max Range: 0.7148. Min Range: -0.7148

Max Act: 0.6333. Min Act: -0.0000

Data Index: 707882 (C4 (Web Text))

Max Activating Token Index: 567

Text #2

Max Range: 0.7148. Min Range: -0.7148

Max Act: 0.5809. Min Act: -0.0000

Data Index: 1128316 (C4 (Web Text))

Max Activating Token Index: 794

Text #3

Max Range: 0.7148. Min Range: -0.7148

Max Act: 0.4699. Min Act: -0.0000

Data Index: 855879 (C4 (Web Text))

Max Activating Token Index: 328

Text #4

Max Range: 0.7148. Min Range: -0.7148

Max Act: 0.4805. Min Act: -0.0000

Data Index: 384718 (C4 (Web Text))

Max Activating Token Index: 207

Text #5

Max Range: 0.7148. Min Range: -0.7148

Max Act: 0.4156. Min Act: -0.0000

Data Index: 835490 (C4 (Web Text))

Max Activating Token Index: 601

Text #6

Max Range: 0.7148. Min Range: -0.7148

Max Act: 0.3888. Min Act: -0.0000

Data Index: 777015 (C4 (Web Text))

Max Activating Token Index: 888

Text #7

Max Range: 0.7148. Min Range: -0.7148

Max Act: 0.4185. Min Act: -0.0000

Data Index: 461114 (C4 (Web Text))

Max Activating Token Index: 123

Text #8

Max Range: 0.7148. Min Range: -0.7148

Max Act: 0.4064. Min Act: -0.0000

Data Index: 1328051 (C4 (Web Text))

Max Activating Token Index: 137

Text #9

Max Range: 0.7148. Min Range: -0.7148

Max Act: 0.4094. Min Act: -0.0000

Data Index: 817029 (C4 (Web Text))

Max Activating Token Index: 1004

Text #10

Max Range: 0.7148. Min Range: -0.7148

Max Act: 0.4044. Min Act: -0.0000

Data Index: 1224501 (C4 (Web Text))

Max Activating Token Index: 509

Text #11

Max Range: 0.7148. Min Range: -0.7148

Max Act: 0.3483. Min Act: -0.0000

Data Index: 327209 (C4 (Web Text))

Max Activating Token Index: 767

Text #12

Max Range: 0.7148. Min Range: -0.7148

Max Act: 0.3683. Min Act: -0.0000

Data Index: 1289543 (C4 (Web Text))

Max Activating Token Index: 130

Text #13

Max Range: 0.7148. Min Range: -0.7148

Max Act: 0.3947. Min Act: -0.0000

Data Index: 1124474 (C4 (Web Text))

Max Activating Token Index: 720

Text #14

Max Range: 0.7148. Min Range: -0.7148

Max Act: 0.3789. Min Act: -0.0000

Data Index: 821810 (C4 (Web Text))

Max Activating Token Index: 891

Text #15

Max Range: 0.7148. Min Range: -0.7148

Max Act: 0.3829. Min Act: -0.0000

Data Index: 811111 (C4 (Web Text))

Max Activating Token Index: 368

Text #16

Max Range: 0.7148. Min Range: -0.7148

Max Act: 0.3656. Min Act: -0.0000

Data Index: 502055 (C4 (Web Text))

Max Activating Token Index: 611

Text #17

Max Range: 0.7148. Min Range: -0.7148

Max Act: 0.3748. Min Act: -0.0000

Data Index: 353285 (C4 (Web Text))

Max Activating Token Index: 795

Text #18

Max Range: 0.7148. Min Range: -0.7148

Max Act: 0.3234. Min Act: -0.0000

Data Index: 1187357 (C4 (Web Text))

Max Activating Token Index: 533

Text #19

Max Range: 0.7148. Min Range: -0.7148

Max Act: 0.3542. Min Act: -0.0000

Data Index: 226162 (C4 (Web Text))

Max Activating Token Index: 550