Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: The Pile

Neuron 3564 in Layer 9

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l-pile')

Text #0

Max Range: 7.3002. Min Range: -7.3002

Max Act: 7.3002. Min Act: -0.0000

Data Index: 1558916 (The Pile)

Max Activating Token Index: 254

Text #1

Max Range: 7.3002. Min Range: -7.3002

Max Act: 6.1481. Min Act: -0.0000

Data Index: 1165914 (The Pile)

Max Activating Token Index: 200

Text #2

Max Range: 7.3002. Min Range: -7.3002

Max Act: 6.4822. Min Act: -0.0000

Data Index: 1223977 (The Pile)

Max Activating Token Index: 386

Text #3

Max Range: 7.3002. Min Range: -7.3002

Max Act: 7.1382. Min Act: -0.0000

Data Index: 1672809 (The Pile)

Max Activating Token Index: 323

Text #4

Max Range: 7.3002. Min Range: -7.3002

Max Act: 6.5403. Min Act: -0.0000

Data Index: 40915 (The Pile)

Max Activating Token Index: 404

Text #5

Max Range: 7.3002. Min Range: -7.3002

Max Act: 6.8098. Min Act: -0.0000

Data Index: 570812 (The Pile)

Max Activating Token Index: 447

Text #6

Max Range: 7.3002. Min Range: -7.3002

Max Act: 6.1699. Min Act: -0.0000

Data Index: 236651 (The Pile)

Max Activating Token Index: 144

Text #7

Max Range: 7.3002. Min Range: -7.3002

Max Act: 6.8724. Min Act: -0.0000

Data Index: 1226856 (The Pile)

Max Activating Token Index: 440

Text #8

Max Range: 7.3002. Min Range: -7.3002

Max Act: 6.5260. Min Act: -0.0000

Data Index: 1210012 (The Pile)

Max Activating Token Index: 433

Text #9

Max Range: 7.3002. Min Range: -7.3002

Max Act: 6.1291. Min Act: -0.0000

Data Index: 149066 (The Pile)

Max Activating Token Index: 280

Text #10

Max Range: 7.3002. Min Range: -7.3002

Max Act: 5.9475. Min Act: -0.0000

Data Index: 1110807 (The Pile)

Max Activating Token Index: 87

Text #11

Max Range: 7.3002. Min Range: -7.3002

Max Act: 6.6078. Min Act: -0.0000

Data Index: 1871516 (The Pile)

Max Activating Token Index: 94

Text #12

Max Range: 7.3002. Min Range: -7.3002

Max Act: 5.8442. Min Act: -0.0000

Data Index: 433120 (The Pile)

Max Activating Token Index: 193

Text #13

Max Range: 7.3002. Min Range: -7.3002

Max Act: 5.8287. Min Act: -0.0000

Data Index: 804454 (The Pile)

Max Activating Token Index: 525

Text #14

Max Range: 7.3002. Min Range: -7.3002

Max Act: 5.9019. Min Act: -0.0000

Data Index: 523564 (The Pile)

Max Activating Token Index: 836

Text #15

Max Range: 7.3002. Min Range: -7.3002

Max Act: 6.1994. Min Act: -0.0000

Data Index: 1345485 (The Pile)

Max Activating Token Index: 596

Text #16

Max Range: 7.3002. Min Range: -7.3002

Max Act: 5.5740. Min Act: -0.0000

Data Index: 1487586 (The Pile)

Max Activating Token Index: 108

Text #17

Max Range: 7.3002. Min Range: -7.3002

Max Act: 5.8947. Min Act: -0.0000

Data Index: 429997 (The Pile)

Max Activating Token Index: 499

Text #18

Max Range: 7.3002. Min Range: -7.3002

Max Act: 5.5307. Min Act: -0.0000

Data Index: 1550542 (The Pile)

Max Activating Token Index: 424

Text #19

Max Range: 7.3002. Min Range: -7.3002

Max Act: 5.9711. Min Act: -0.0000

Data Index: 122696 (The Pile)

Max Activating Token Index: 310